It would get rid of a whole bunch since that is where most of them are. Wash your hands after.
They can also get into the fruit and can cause dropping or interior browning, gross. It also says something about how hard it is to kill them, they get into places contact sprays can't... so something that absorbs into the plant like Forbid and kills them when they feed is best. Those neem extracts like azamax and azatrol can be watered in to become systemic, it would cost much more I am sure and take longer but I bet it would work better than usual in self watering containers.