Yes Remove about Half off,as the tree is too young and not well adapted to East Coast cold climate.
Mine also suffer aether from Stress or FMV,whatever it is,the tree is not doing very well.
A lot of fruits now like in your case mean death of the plant next winter.