I went from a 2001 330Ci 5 speed to a 2005 M3 6 speed & 1997 A4 Auto (daily driver) to a 2001 S4 6 speed.
My comparison doesn't really compare well since both my non-M and M BMW were newer than my non-S and S Audi, but it does take a look at the earliest and latest years of both the E46 and B5 chassis.
First off, there was nothing I liked about my 97 A4 except that it had 4 doors. It was slow, sluggish, non-responsive and troublesome. I can't even compare my 05 M3 to the 01 S4 because it's completely out of it's league so what I'll do is compare my 01 330Ci to my 01 S4. Right off the bat, my 330 steering was more responsive, the suspension was tighter, the clutch engaged smoother, the engine was quieter, the engine and trans mounts never went bad even at 140k miles vs the S4's 60k, who's trans rocks between shifts.. The only aspect I like about my S4 is the power and AWD. Other than that, it's simply an understeering 4 door family sedan with twin-turbos. Looks wise, the E46 out-shines the B5 any day of the week. As for the newer B6, 7and 8s vs E90 and 92s, the B7 wins, but will never beat the E46.
I miss my M3 soo much. I do feel more grown up driving the S4 around, but I don't think Audi's "S" will ever have the status symbol that BMW's "M" carries. Plus, I like being a kid so M POWER all day!