Hmm, that is a tough one.
If it isn't productive and not really growing 3 years later, I would dig it up and replace it with a new tree. Maybe it has some some sort of insect affliction or disease. You never know, it's possible there is something in the ground at that one location affecting the tree. Either way, if you dig it up you may discover something; and if you plant a new tree and it grows normal...bingo, your ahead of the game. 3 years does seem to be the magical fig tree number around here, meaning if you don't like what your tree is making or doing after 3 years, you get rid of it.
Some pictures are always helpful though.