I Live close to you and have the same problem with Celeste.
A partial fertile cultivar will drop fruits,in climate that are ,not ideal and this is the case ,in our location.
People in other states with better climate have no problem like this.
My advice:Get Improved Celeste(ORN),if you really want a Celeste type or get other cultivar like Takoma Violet,Malta Black,Saint Anthony, Adriatic jh,and you will not have that problem any more.