I will have to try x264 some more. Initial tests show quality is better than XviD but not as good as Nero Recode (which as I said earlier, I can't detect any difference from the original).
The MPEG-4 Part 10 spec is a lot stricter than the old MPEG-4 Part 2 spec. Generally if a codec supports all the profiles available (basically if it supports High Profile) then final output quality will be very, very, close to identical. That's the whole idea of the standard.
Nero Digital AVC, VC-1 (Microsoft's version), and x264 all have the same video quality if the same settings, features, and bitrates are used.
The implementations can certainly be better or worse of course. Not in quality, but in ease of use, speed, and capabilities. There can also be differences in resizers (if you do that) and pre-processing that can create apparent differences in quality. This isn't the fault of the codec though, it'd be the support applications (and generally, with the right stuff, the open source stuff is up-to-snuff with Nero's). But, I'm not disputing (in any way) Nero Digital's good results! Just the idea that (admittedly not fully investigated) x264 is inferior in quality. They're on par with each other.
Generally, x264 is considered (right now) faster on most hardware (and free). Nero Digital is, of course, much easier to use (and not free). That said... I do like (and own) Nero. Great stuff, and you'll certainly be very well served by it if that's what you like! The software isn't quite as flexible as MeGUI and it's army of command line utilities, but sometimes that's a very good thing (to keep your head full of hair).
Same can't be said for Quicktime at all. Unfortunately, Apple didn't support all of the MPEG-4 AVC Profiles, and didn't even fully support h264 properly. It looks pretty good, but it isn't as good as Nero, VC-1, or x264.