If I was you I would remove all young fruits and leave less than 2 dozen,for this year.
1. quality of fruit will improve
2.Your tree will not be dead next Spring,from overwork,and if alive will be ill and no fruits next year.
yet I want to congratulate you for the care you give to your tree.
There is a reason,behind the success,and has to do with you.