Yes they will improve,every year till they are 7 years old and that is their top taste.
But of course if in some years you have 20 inches of rain in July and August ,in those years the fruits will not be at their best quality and taste will suffer.
Yet again ,an older tree will make ,more satisfactory fruits,in those bad years,one reason is because the fruits will be growing at much higher distance from the soil line and so,it can fight the rain better,and the wind will dry the rain off the fruits faster,and that will make a difference.