Most figs can take to 25 degrees, so I leave mine out till all leaves fall off, then into the garage. I'm in zone 6a. You babied yours, I have seedlings outside it's 53F and they don't have true leaves yet. no need to bring it in, you want it tough. If mine can't survive the treatment, I don't want them. So far all have. Sure you have to acclimate them, but they need to be ready in a few months to withstand 25F, so mine never see the house except in the winter when I'm rooting cuttings, once spring hits, they will never enter the house again. I will take them in and out for a week or two, no more. Slowly exposing them to the sun and wind etc.
Yeah i wanted to have about 6, sounded good, I have 36 now. I am though eliminating about 16 of them. Keeping what I like best.