There are other leaves that are fine the ones turning yellow arw the ones damaged the most by the rust. I'm was just a little shocked how quickly it accelerated this weekend.

Hi 71GTO,
not being a smarty pants here ok ?
In short you do not need to worry.
In colder area's as i have no experience growing in Florida for example.

They do this every year, leafs get the so-called rust spots look like crap , yellow fall off. Some leaves on same tree will not be as bad if at all thats how it is.
Each late season weather can somewhat be different with cold weather arriving earlier or later perhaps.

Some seasons here near Chicago plants go in storage with leaves on some years with no leaves its the weather that dictates this and how the leaves will look, some years leaves look not so bad some years like crapola.

This season here colder weather arrived earlier with much more humidity and rain as of late .
I have not taken pictures showing how crappy the leaves look but im sure they look just as nasty as yours and some still look real good .

Thanks Martin, I'll calm down. This is My first whole year. Lots of things are still new to me.

Ah that explains your concerns i did not recall you were basically new to growing them.

I have fig rust on some of my trees as well because of the rain we had. It's fungal and so you don't need to do anything about it. It won't harm the tree. When your trees go dormant and the leaves fall, just make sure you pick up the leaves and discard them. I felt the same way you did when I first started doing this. Ironically, some varieties are side by side with the ones with the rust and don't seem to be affected by it.