I have couple potted fig trees and one in ground fig tree having tiny figlets, I think that you should check what type of fig trees are they, I have Coll de Dama tree with figlets, It will drop all breba, so I removed all of them.
Calvin is right that you should really watch out of magic number of 38F/3.3C, I bought "Desert king" tree with figlets/young leaves at March last year, Although temps got above 50F/10C day, Temps got too low at right 38F/3.3C, fig tree was shriveling and dropped all leaves and figs, cold shocked fig tree, It took 2 months for it to recover. This year, I bought a big/cheap simple Dial Temperature meter, I install it outdoor and I can see it inside my window, I will pay more attention for those cold night this year.
For my in ground fig tree, It is about 15FT X15Ft, I covered its ground with plastic sheet to heat up ground with Sun, Hoping that it can tolerance cold night.