Anthony:Growing in pot is more difficult for the tree because it get stressed by certain hot days,may be one day it needed water and it did not get it,lack of nutrients ,etc.
If the tree produces good harvest in ground where mother tree is,but you can't put yours ,in ground,then bury the pot 3 inches in the yard.
Make lateral drainage holes on bottom side of pot and plug bottom holes.
That way you cut roots off in December and take plant inside.