Most tests seem to be compairing sound quality when you rip them to CD and then listen over large speakers and not headphones.
The only “real” test I’ve read was one conducted by Sound & Vision several years ago. In that test, they were actually comparing MP3 vs AAC vs WMA. They brought in a trained listener from each of the codec creators plus the author of the article and conducted a double-blind test. All the trained listeners were able to tell the original material from the compressed versions and three of the four were able to accurately identify each of the codecs. Surprisingly, it was one of the trained listeners who couldn’t identify the individual codecs.
I can’t recall the exact bitrates tested, I think it was at least 128K. I do recall that they used headphones and IIRC they used over-the-ear electrostatics, similar to these Sennheisers.
I also recall that the material was chosen because it was know to cause specific encoding artifacts with each of the codecs. For example Pearl Jam’s Daughter was chosen because the interactions between the vocals and the acoustic guitar at the start of the piece are know to cause problems for codecs.
Personally,I can almost always identify mp3s. MP3 encoders add a 3db bass boost and because I really hate overboosted bass, I tend notice it.
The artifact that most people can hear is distortion on cymbals. Instead of being a nice clean ringing sound, the encoder adds a bunch of noise and you get a sound which some people describe as “frying bacon”. I’ve also noticed a similar problem with The Who’s Love Reign O’er Me. The rain at the start of that song also ends up sounding like frying bacon even at high encoding rates (I’ve tried 320K CBR and can still hear it).
So, yeah encoding artifacts are real and can be heard, especially by trained listeners. However, at high-bit rates you’ll be really hard pressed to hear anything on the vast majority of music out there. Every now and then you’ll run across a piece of music that the encoder can’t handle and generates a bunch of nasty, easily heard artifacts but they’re really rare.