First of all, once the leaves are yellow and fall off easily, it doesn't matter whether you brush them off or not. The damage is done, and the leaf is a goner already. You need to address the issue first. I'm not sure how big your pots are, but I'd suggest up-potting. I had a number of plants showing the same symptoms as you describe, and I was watering them quite a bit. Seemed like it was hard to keep up with water loss due to evaporation and transpiration. I put them in bigger pots, still with well draining soil, and it seemed to correct the issue.
I have this theory that if you have roots circling the pot, even if the pot is huge, a large proportion of the root mass can be in the outer inch or two of the pot. This part heats up and dries out quickly, so even in a very large pot you can have issues with not enough water if it's a hot day, particularly in a black pot.