I have about 10 different potted figs and the leaves get brown specks then yelllow and fall off. I kept them in part shade so the roots wouldn't get too hot when the temps were in the 90's. once I plant them in the ground they grow great and don't have the problem like the potted ones do. They have only been potted about two months since buying them in 4" pots from edible landscapes. I watered them once a week.

It is kinda what figs do, LOL

small pot= less water+too hot+less food
they starve and they turn yellow to show unhappiness.

I have limited experience but I'd say more water. Small pot are easy to pick up to judge the amount of water left in the soil.

Looks like fig rust to me. Not much you can do. I have it too. I'm going to use lime-sulfur in all seasons oil once dormant to kill any spores that try to overwinter. Lime-sulfur is hard to obtain. You could use copper to try to kill any spores. If worried about it spray copper next year too. Once the plant has it, you can only slow it down with copper, not stop it. Copper will prevent a clean tree from getting it though.
That said, your tree will not die. It is hurting it though. I have had figs aborted because of lack of nutrients that the leaves supply. They tend to ripen prematurely.
http://www.gardeningknowhow.com/edible/fruits/figs/figs-disease-rust.htm