My advice would be no for now. However, you should get your soil tested by your Ag agent to verify what I am going to tell you.
Since you have clay soil, it is likely alkaline in nature (high pH). Clay soil also tends to be high in Phosphorus. Problem with high Phosphorus levels is that they inhibit the uptake of Nitrogen by plant roots (especiallly in young trees). Of you add that phosphate supplement to the soil, you will likely upset the balance you have now and make the Nitrogen uptake more difficult for your trees. Besides that, Phosphorus is the nutrient that fig trees need the least of (and it's already in abundance in your clay soil anyway!). One way you can verify this is by recalling/observing if you have ever noticed the presence of Iron Chlorosis in some of your plants? If so, you probably have very alkaline soil (like I do) and what I have told you is correct. More phosphorus would be exactly the wrong thing to add.
Since the trees have borne adequate fruit in the recent past and since they are all still young, the problem is likely rooted in some other source.
Here is more info on Iron Chlorosis and it's management: http://forestry.usu.edu/htm/city-and-town/tree-care/preventing-and-treating-iron-chlorosis-in-trees-and-shrubs