Hi,
I would let 4 or 6 stems grow on the trunk, and shorten the tree. I would select stems at 1' or 2' of height.
I don't want to use a ladder for fruit picking.
I would try to clean/remove the area where the nest was.
Of course, if you're in a hot climate, and use the tree for shade, you may want to keep some parts of the canopy a bit longer.