Living in the northeast all my life I have to say, YES, be concerned. You need to grind it all off and get rid of it. Rust will never stop spreading once started.
Properly done, you can stop it. Sand, grind, use some good anti-rust and paint and seal. But get rid of it now before it's to late.
They do sell some good stuff that will turn it rock hard and black, but I don't remember the name of hand, but it works good and will not spread. Still, best way is to grind it off and treat it and paint.