I found this a very interesting thread and given I have had plenty of experience of an RS4, owned an e92 M3 and currently own a C63 with performance pack, I guess I'd better give my humble opinion.
The RS4 is a much older design than the C63 and M3 and this is particularly noticeable on the inside of the car which although very well built, is quite dated in design these days.
However, the RS4 is easily as well built as the C63 and much better than the M3 which rattles and creaks far too much for my liking. The RS4 also has exceptional roadholding, on the road at least! Like has already been said, it really does just grip and go round corners. But it isn't 'fun' in the way the M3, and, to a lesser extent, the C63 is.
However, the main reason I've never been tempted by the RS4 was simply because it just doesn't have the performance of the M3, never mind the C63 which is in another league again! I guess it's the 4WD powertrain that saps power (although it also has a reputation of not making anywhere near the power that Audi quote) but it just is not as quick as the other two (unless it's going round wet roundabouts)!
Ultimately, of the three cars, the M3 handles the best and is the only choice for regular track days. The Audi has the most mechanical roadholding (and the C63 the least by some margin even with the performance pack) so if you want to go fastest in the wet then this car is for you. The C63 has the most performance, the best sound and also the nicest combination of interior design and build quality.
All IMO of course!