PY, welcome to the forum.
A lot of my breba figlets shriveled up this spring because they formed indoors, and I suspect they didn't have enough light or heat.
Some varieties, like improved Celeste, are known to drop small figs if they are deprived of water.
For this reason, I planted mine in ground this year, to allow the plant to grab its own water from the soil as it needs it.
I don't know if this is true, but I heard/read somewhere that fig embryos should not be touched, or they abort. Maybe someone will confirm or deny this for sure.