Yes, if it is really a rust infection, then remove those leaves.  (But not all brown spots are really rust).  Don't compost leaves with rust.  

Mike 

Thanks again.    Such a   b e a u t i f u l  day !

Yes no harm at all to plants.

Soni, I not only remove leaves that look like they may have rust, but I also spray the plant, the top of the soil, and the pot, with a spray for rust. Then apply enough live compost to the soil surface to completely cover the top of the soil. Rust can be difficult to remove once it gets on your plants.

When spring comes do not place pots back in the same spot, until you have sprayed the area.

Bob @ T. Pine Connecticut - zone 5b/6a 

  Thanks Bob,  Yikes! I better get busy.

Soni.  did you have frost this morning?

No frost this morning, but it was 35F at 7:15am.   How was it at your place?  I forsee a problem with the greenhouse keeping too warm by day.  Its on a covered porch but the exposed side faces Soiuth,  I need to cover it or whitewash that side  ,,  something

We had a pretty good frost this a.m.

Mike, I use Immunox.

Bob @ T. Pine