I'm not familiar with your growing conditions, but as a general rule you need about 80 days for a fig to ripen from the time it is formed. If you don't have that much time left, yes remove the smaller ones. They will not ripen in time and do take energy from your tree.