Another day, another anonymous poster on the Internet criticizing a reviewer's test results because he doesn't agree with the conclusions. In this case, it's in a comments thread on Lauren Dragan's mega-review of under-$30 in-ear headphones for The Wirecutter. I was one of the listening panelists for this test. The anonymous poster was criticizing our evaluation of the Monoprice 8320, which is the darling of headphone enthusiasts -- or at least of headphone enthusiasts who spend a lot of time posting anonymous comments on the Internet.
We criticized the 8320 for sounding too bright and lacking bass. The anonymous poster argued that the 8320 is an "accurate" headphone, and because we included it in a test with what he called "a bunch of colored headphones," the Monoprice suffered in the comparison. Unfortunately, I let myself get pulled into using his term -- "accurate" -- which I've for the most part abandoned in favor of the term "neutral," which means that a product doesn't have any noteworthy emphasis or de-emphasis of any particular part of the audio spectrum -- i.e., bass, midrange or treble.
The problem with calling an audio product "accurate" is that it's difficult to know what "accurate" is. Technically, an accurate audio product should replicate the sound of whatever it's being used to reproduce: a human voice, a violin, a car crash, etc. But how do we know what that original sound was like?
Dr. Floyd Toole addresses this problem in his book Sound Reproduction: The Acoustics and Psychoacoustics of Loudspeakers and Rooms, in which he discusses the "Circle of Confusion." Here's the circle:
We evaluate products such as headphones and speakers by using recordings, which are made by monitoring on headphones or speakers. Those headphones and speakers are evaluated by listening to recordings that are monitored on headphones or speakers that are evaluated using recordings ... etc., etc., etc.
Got it? Toole's point is that we have no way of knowing if the sound we're getting from a speaker or headphone is the "right" sound unless we know what the recording we're using for the evaluation is supposed to sound like. And let's not forget that vocals are recorded with microphones that have significant peaks and dips in their frequency response; accompanied by electric or electronic instruments; equalized using tone controls; run through various sound processing devices; then run through another round of EQ and processing during the mastering process. Even audiophile recordings are almost always produced with some amount of audio processing -- and of course, even with the purest recordings, the choices of microphones, mic positions and venue have a huge effect on the sound.
Thus, we're using inaccurate recordings to judge the accuracy of an audio component.
So how do you know what would represent an accurate portrayal of that recording? You don't.
Toole makes his point to stress the importance of developing measurements that can tell us whether an audio product is delivering something that a majority of listeners will perceive as accurate. During decades of work at the Canadian National Research Council and Harman International, Toole and his colleagues developed a way of measuring a speaker that could predict whether or not the speaker would be perceived as accurate. You can hear the results of that work in speakers like the Revel Performa3 F208. Or the F206, which I purchased to use as a reference speaker.
Sadly, we're not yet at the point where measurements can tell us exactly what an "accurate" headphone is. Existing methods of headphone measurement, detailed in the IEC 60268-7 standard, are, in the words of a manufacturer I recently spoke with, "woefully inadequate." Most measurements now used for headphones were developed for use in the hearing aid industry, where speech reproduction, not music reproduction, is the goal -- a problem reflected in the fact that existing headphone measurement gear isn't designed to deliver reliable results at frequencies above 10 kHz. Even one of the most basic ideas of headphone voicing -- the diffuse-field equalization, meant to simulate the sound of speakers in a room -- has been questioned by Toole's former research colleagues at Harman, Sean Olive and Todd Welti.
I've measured more than 170 headphones at last count, and conducted tests of most of those headphones using multiple listeners. Even with all that experience, I can gather only a rough idea from my measurements about how listeners will perceive a headphone's sound quality. Even worse, the results from listening tests -- especially, in my experience, with in-ear headphones -- seem to be less consistent with headphones than they are with speakers.
But I can tell you this: Headphones with treble that "takes your head off," such as the Monoprice 8320, or the Grado SR225 that I own, sound nothing like the Revels in my listening room. Nothing like the B&W CM10 I just tested for Sound & Vision. Nothing like the Music Hall Marimba minispeaker that did so well in the minispeaker face-off I did for S&V. These headphones -- revered by some enthusiasts for their elevated treble, which these enthusiasts mistake for detail, which they in turn confuse with accuracy -- do not present a faithful reproduction of music.
A good way to confirm this would be to buy the 8320 (it's only $7.65), take it to any recording studio (home or professional), and compare the sound of the studio monitors with what you hear in the headphones. What you hear through the headphones will sound radically different. And thus not accurate.
Researchers such as Olive and Welti, and manufacturers such as Paul Barton of PSB, are devoting a great deal of time and resources to figure out what, exactly, an "accurate" headphone is. Until they make further strides in that work, anyone who describes a headphone as "accurate" (or "reference" or whatever other meaningless phrase they want to attach) is overestimating our understanding of headphones.
Meanwhile, if you want to enjoy the benefits of work done by researchers such as Toole, Olive and Welti, read "A Simple, Cheap Way to Good Room Acoustics."
Image of AKG headphone by Brent Butterworth