I generally consider worms in soil or pots a very good thing. They will aerate the soil as well as help decompose uncomposted items and their castings have beneficial properties. Most of the other insects, including millipedes, generally are good or neutral as they mostly feed on dead vegetation.
Fungus gnats can feed on living roots and introduce infective fungus to living tissue, thereby possibly compromising a young cutting. However, I had a severe fungus gnat infestation in my cuttings a couple years ago, which likely did kill some/many. But many survived, and once they were in one gallon pots and placed outside, the problem went away. So I doubt they could do much to a live, healthy tree grown outside.
Some grubs feed on living roots. But again, I doubt this would significantly damage a healthy tree. Just another form of root pruning. Unless there are dozens of them in one pot.
Even if you were to do your best to rinse all the soil out of the roots and put in fresh, sterile soil, there could still be eggs of these things attached to the roots that would hatch once you put in the new soil. Even if not, eventually they would find their way into the pot if they are living in the ground, no matter how hard you try to keep them out.
For the blueberry, maybe the pH of the mix is too high. They like very acidic soil.
For the figs, if you are in SoCal, growing in the ground would be preferable, but perhaps you are not able to do so. It's likely that they need a bigger pot, or perhaps there is some other issue with the growing media. But it's very unlikely that it's caused by the creepy crawlies.