I remove old sealant and and wax with the Chemical Guys Citrus Wash
. Dish soaps do remove them but some people have reported with repeated use it dried out vinyl and rubber trim pieces.
Any decent polish with good arm pressure or a buffer would remove the old sealant or wax so I doubt that was the cause. I would try to polish the area again. Also what products (polish, pad, buffer vs hand) are you using?
The M205 and a white pad should be able to remove a lot of the buffer trails and is one of my go to combinations. If that doesn't do it or if that combination leaves too much behind try using the ultra light finishing polish Menzerna Micro Polish SF 4500 (PO85RD)
with a black pad afterward.
I'm surprised to hear the M105 and M205 didn't remove the majority of swirls. I typically use at least two orange pads with the M105 and two white pads with the M205 afterward. Use about 10lbs of pressure and turn the PC up to speed 6, move slow and steady for best results. if you need more power try the Surbuf R Series MicroFingers Buffing Pads
and the M105 that should do it!
Greg @ DI