Fruit production requires significant energy, it saps energy reserves in your plant, redirecting crucial energy to places it shouldn't be at that age/size. It takes a good amount of energy for an actual tree to produce fruit, much less a twig, and a half gallon is pretty small. To boot, first-year fruit tends to be pretty worthless, and (at least with my Italian Honey trees) the breba crop is really not that great already.
I'd always encourage knocking off fruit on something that small so the maximum energy goes towards building up that trunk and vascular system.
I've stunted the growth of far too many fruit trees and fruit bushes being stubborn and leaving fruit on. I almost killed two of my blueberry bushes a couple years back because I had to have fruit the first year in-ground. I picked off the other three bushes of fruit (planted same year). That was four years ago. The bushes I plucked fruit off of are 3'x3' in size and well branched. The two I left the fruit on are still less than a dozen twigs thinner than a pencil and hardly a foot tall or wide.
Just sharing my experience.