I have a "Desert King" green fig in a 24-inch pot, where it has lived for years. It put out a nice early crop back in July and now it's on its second crop -- but for some reason, a lot of the figs are getting little spots on them and falling off, or I can just grab the little figs and pull slightly and they come loose. The stem connection point on the fig has a browninsh ring around it. This also happened last year. I water the plants daily and feed occasionally with bone meal. A photo of one I just removed:

It has lost quite a few fruits this way, and I'm wondering if this could be fungal, or maybe a nutrient deficiency. My "Petite Negra" black fig, ten feet away from this plant, is fine. Anyone know what this problem might be?