Noisy Decisions
I have tinnitus. When there isn’t much sound in my environment, for me it still isn’t quiet. I hear a constant background hum. It is hard to describe what this noise sounds like. I’ve lived with it for too long. Remembering back to when I first noticed it, I thought there was some nearby electrical device humming. Was it my phone plugged into the wall outlet? Or??? I remember getting up from bed to hunt for the source of that noise.
I can’t forget that noise or ignore it. It doesn’t go away. But it doesn’t dominate my headspace. I’ve learned to slip between that noise and my desire to sleep or to just be in a quiet place, and not let it distract me. I’ve learned to deal with tinnitus.
Recently I finished reading Noise: A Flaw in Human Judgment by Daniel Kahneman, Olivier Sibony, and Cass Sunstein.
The entire book is about the “noise” in human judgments and what we can do to lessen its effects. So what exactly is this noise? A simple definition is “noise” is undesirable variability in judgments. Call this system noise if you will.
Both recurring and onetime decisions are influenced by noise. Depending on the time of day, how well I slept last night, what others say, and even how we as a group decide how to decide effects my judgment. This noise, in addition to any biases I have, affects all my judgments.
Kahneman, Sibony and Sunstein introduce two different types of system noise: level noise and pattern noise. Let’s consider each in turn.
Level noise is easiest to understand. It is the variability in the average level of judgments by different people. People judge on different scales. Consider rating a talk at a conference. Perhaps you never give a conference speaker the highest possible rating because you believe they could do better. Or, maybe if you are star struck, you always rank a presentation from a well-known speaker more highly. Personally, I know that I tend to not rate speakers either as very high or very low, because, well…I’m sort of middling with my ratings. On average, humans aren’t average in their judgments.
The other kind of noise, pattern noise, is often an even bigger factor in our judgments. It is comprised two parts: occasion noise and our own personal idiosyncratic tics. Occasion noise is the variability in the judgment at different points in time. Depending on my mood, how stressful the situation, how well I slept last night, or how the question is put to me, my judgment will vary. A simple example of occasion noise that software folks can relate to is estimating how long it will take to complete a task. My mind isn’t the same today as it was yesterday. Heck, from moment to moment, I might give a different answer simply because I am thinking about the task differently, or I that I am hungry (and hence tend to come to a snap judgment), or I’m grumpy, or I’m happy.
The second source of pattern noise is our personal attitudes toward the particular judgment context. Consider, for example, this kind of noise when reviewing conference proposals for papers or talks. Some reviewers are harsher in their personal rating for some proposals and more lenient in others. This variability reflects a complex pattern in the individual attitudes of reviewers toward particular proposals. For example, one person may be relatively generous in their review of proposals on a particular topic. Another may be particularly keen on proposals that seem to break new ground but be a harsher judge of proposals on topics that are perceived to cover familiar territory.
As Kahneman, Sibony, and Sunstein state: “Noise in individual judgment is bad enough. But group decision making adds another layer to the problem. Groups can go in all sorts of directions depending in part on factors that should be irrelevant. Who speaks first, who speaks last, who speaks with confidence, who is wearing black, who is seated next to whom, who smiles or frowns or gestures at the right moment—all these factors, and many more, affect outcomes.”
Having been part of many conference review teams as well as on the receiving end of their reviews over the years…I find that the dynamics of group decisions to be a particularly salient example of system noise.
The information about system noise in general and noise in group decision making can be rather depressing. If we humans are naturally wired to be imperfect and flawed in our judgments, how can we hope to make reasonable decisions? And, once we become aware of our judgment errors, and try to be better decision-makers, the actions required to lessen the effects of noise these authors suggest seem surprisingly difficult to carry out.
Awareness is a good first step. But it’s not enough. In contrast to tinnitus, of which I’m constantly aware, the noise in our judgments is at first difficult to perceive. But once you become aware of sources of system noise, you start noticing them everywhere. How and when (and even whether) it is appropriate or feasible to mitigate these sources of noise is a topic for another day.