Chivas: One Hundred percent I agree with you.
Also the larger the fruit the more is dependent on full sun.
In cold climates,Zone 6 and 7,fruits will not form early in Spring,if it is a cold Spring,but can be forced to form early by insistent pinching,and placing pot in full sun,or have the plant ,planted in full sun.
Only if one manage to make the tree form early fruits ,they will ripe in cold zones,as mentioned.
Of course a greenhouse will help in Spring to force plant form fruits inside then take out plant in May and keep growing.