1. Technology
You can opt-out at any time. Please refer to our privacy policy for contact information.
Brent Butterworth

Who's to Say What "Accurate" Is?

By December 4, 2013

Follow me on:

AKG question markAnother day, another anonymous poster on the Internet criticizing a reviewer's test results because he doesn't agree with the conclusions. In this case, it's in a comments thread on Lauren Dragan's mega-review of under-$30 in-ear headphones for The Wirecutter. I was one of the listening panelists for this test. The anonymous poster was criticizing our evaluation of the Monoprice 8320, which is the darling of headphone enthusiasts -- or at least of headphone enthusiasts who spend a lot of time posting anonymous comments on the Internet.

We criticized the 8320 for sounding too bright and lacking bass. The anonymous poster argued that the 8320 is an "accurate" headphone, and because we included it in a test with what he called "a bunch of colored headphones," the Monoprice suffered in the comparison. Unfortunately, I let myself get pulled into using his term -- "accurate" -- which I've for the most part abandoned in favor of the term "neutral," which means that a product doesn't have any noteworthy emphasis or de-emphasis of any particular part of the audio spectrum -- i.e., bass, midrange or treble.

The problem with calling an audio product "accurate" is that it's difficult to know what "accurate" is. Technically, an accurate audio product should replicate the sound of whatever it's being used to reproduce: a human voice, a violin, a car crash, etc. But how do we know what that original sound was like?

Dr. Floyd Toole addresses this problem in his book Sound Reproduction: The Acoustics and Psychoacoustics of Loudspeakers and Rooms, in which he discusses the "Circle of Confusion." Here's the circle:

We evaluate products such as headphones and speakers by using recordings, which are made by monitoring on headphones or speakers. Those headphones and speakers are evaluated by listening to recordings that are monitored on headphones or speakers that are evaluated using recordings ... etc., etc., etc.

Got it? Toole's point is that we have no way of knowing if the sound we're getting from a speaker or headphone is the "right" sound unless we know what the recording we're using for the evaluation is supposed to sound like. And let's not forget that vocals are recorded with microphones that have significant peaks and dips in their frequency response; accompanied by electric or electronic instruments; equalized using tone controls; run through various sound processing devices; then run through another round of EQ and processing during the mastering process. Even audiophile recordings are almost always produced with some amount of audio processing -- and of course, even with the purest recordings, the choices of microphones, mic positions and venue have a huge effect on the sound.

Thus, we're using inaccurate recordings to judge the accuracy of an audio component.

So how do you know what would represent an accurate portrayal of that recording? You don't.

Toole makes his point to stress the importance of developing measurements that can tell us whether an audio product is delivering something that a majority of listeners will perceive as accurate. During decades of work at the Canadian National Research Council and Harman International, Toole and his colleagues developed a way of measuring a speaker that could predict whether or not the speaker would be perceived as accurate. You can hear the results of that work in speakers like the Revel Performa3 F208. Or the F206, which I purchased to use as a reference speaker.

Sadly, we're not yet at the point where measurements can tell us exactly what an "accurate" headphone is. Existing methods of headphone measurement, detailed in the IEC 60268-7 standard, are, in the words of a manufacturer I recently spoke with, "woefully inadequate." Most measurements now used for headphones were developed for use in the hearing aid industry, where speech reproduction, not music reproduction, is the goal -- a problem reflected in the fact that existing headphone measurement gear isn't designed to deliver reliable results at frequencies above 10 kHz. Even one of the most basic ideas of headphone voicing -- the diffuse-field equalization, meant to simulate the sound of speakers in a room -- has been questioned by Toole's former research colleagues at Harman, Sean Olive and Todd Welti.

I've measured more than 170 headphones at last count, and conducted tests of most of those headphones using multiple listeners. Even with all that experience, I can gather only a rough idea from my measurements about how listeners will perceive a headphone's sound quality. Even worse, the results from listening tests -- especially, in my experience, with in-ear headphones -- seem to be less consistent with headphones than they are with speakers.

But I can tell you this: Headphones with treble that "takes your head off," such as the Monoprice 8320, or the Grado SR225 that I own, sound nothing like the Revels in my listening room. Nothing like the B&W CM10 I just tested for Sound & Vision. Nothing like the Music Hall Marimba minispeaker that did so well in the minispeaker face-off I did for S&V. These headphones -- revered by some enthusiasts for their elevated treble, which these enthusiasts mistake for detail, which they in turn confuse with accuracy -- do not present a faithful reproduction of music.

A good way to confirm this would be to buy the 8320 (it's only $7.65), take it to any recording studio (home or professional), and compare the sound of the studio monitors with what you hear in the headphones. What you hear through the headphones will sound radically different. And thus not accurate.

Researchers such as Olive and Welti, and manufacturers such as Paul Barton of PSB, are devoting a great deal of time and resources to figure out what, exactly, an "accurate" headphone is. Until they make further strides in that work, anyone who describes a headphone as "accurate" (or "reference" or whatever other meaningless phrase they want to attach) is overestimating our understanding of headphones.

Meanwhile, if you want to enjoy the benefits of work done by researchers such as Toole, Olive and Welti, read "A Simple, Cheap Way to Good Room Acoustics."

And if you're curious about that white AKG K545 headphone in the photo, read my review and measurements.

Image of AKG headphone by Brent Butterworth





December 4, 2013 at 10:34 pm
(1) TWelti says:

Hi Brent, I agree wholeheartedly . Your comment about confusing excessive treble with “detail” makes me wonder what other characteristics , given such names as “silky”, “strained”, “focused”, etc. and (my personal favorite: “chocolatey”), are simply spectral imbalances.

December 5, 2013 at 4:00 pm
(2) George Hilbert says:

Notwithstanding all the other factors in the article, “accurate” is rather easy to quantify. In terms of a component, amplifier, speaker or headphone, it means that the waveform measured at the input could overlay the waveform at the output, as measured by O-scope coupled with lab grade pickups or other suitable lab equipment. Any “coloring” would change the relative amplitude or frequency of the waveform and be easily spotted.

All the stuff done to the sound prior to entering the component is irrelevant. There are a lot of things that can be argued about sound, but “accuracy” is hardly one of them

December 5, 2013 at 7:54 pm
(3) Brent Butterworth says:

George, there’s one problem with what you wrote. Yes, that’s true of electronics. But once we get to transducers (i.e., speakers and headphones), we run into all sorts of problems.

That “lab grade pickup” you mention would in the case of transducers be a microphone of some sort. In the case of a headphone, the microphone is typically in the form of an ear/cheek simulator or head/torso simulator. That device has its own resonating chambers, intended to mimic the human ear. The parameters of that device are determined by IEC standards according to an average of human ear responses. This standard was chosen by a committee of fallible human beings using their best judgment as to what would be accurate. Some manufacturers I’ve spoken with don’t believe these standards are valid; they measure using a microphone mounted in a flat plate.

Then a correction curve is typically applied to simulate the effects of different positions of the transducer under test. Using no curve will give you a drum reference point (DRP) measurement, equivalent to if the transducer were right at the eardrum and the ear canal and pinna were not there. You could use a curve for ear entrance point (EEP), roughly the point where your ear canal terminates, or ear reference point (ERP), roughly the point where the central axis of your ear canal would meet your palm if you pressed your hand against your ear. These compensation curves are established up to only 10 kHz, partly because, as I pointed out, they were developed for hearing aid testing, and partly because there are such colossal variances in ear canal resonance from human to human above 10 kHz that it’s questionable whether a standard for those frequencies could ever be set.

Then you might add a correction curve for diffuse-field equalization, another IEC standard that is meant to mimic the sound of speakers in a room. But as I pointed out in the article, the diffuse-field EQ curve has been questioned and to some extent disproven by audio researchers. Some people who measure headphones incorporate the diffuse-field curve into their measurements. Some don’t because they don’t believe the diffuse-field curve is valid. (I don’t mainly because, based on my conversations with the researchers involved in this field, I expect a new target curve will be established in the not-too-distant future.)

Then in the case of speakers, where do you place that measurement microphone? On-axis with the tweeter? Between the tweeter and the midwoofer? At what angles do you measure? Do you average a bunch of measurements taken at different angles? How do you weight those different measurements? All the same, or do some count more than others?

These are questions on which very qualified and experienced experts disagree. There is nothing simple about determining the accuracy of a speaker or headphone through measurements.

Leave a Comment

Line and paragraph breaks are automatic. Some HTML allowed: <a href="" title="">, <b>, <i>, <strike>

©2014 About.com. All rights reserved.