Soliciting feedback from many different stakeholders is well-known to be valuable to firms as input to their own improvement processes. Restrooms at airports have smiley-face buttons that translate to numbers on the net promoter scale, and it’s easy to imagine how that one button press is contextualized: Time of day, day of week, month, year, flights recently arrived and soon to depart, cleaning staff on duty, time since last cleaning, and so on. Likewise, when we partner with publishers and large non-profits at Alley, we are constantly looking to crystallize feedback from our stakeholders and from their audiences into smart, well-considered digital products.
It's helpful to look at these feedback processes as group judgments. We can even consider button-pushing efforts of hundreds of air travelers form a sort of group judgment. In their book Noise: A Flaw in Human Judgment, the behavioral economists Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein make a strong case for group estimation as a noise-reduction technique, but caution that many methods of group estimation are perilous and can lead to noise amplification. For example, one loud person with a strongly-held opinion can sway a group.
Kahneman, Sibony, and Sunstein offer some clear guidance to reducing bias and noise in group judgments:
- Give the group several structured subjudgments to make rather than one big judgment
- Put independent thinking first in a group discussion context to prevent anchoring
- Make the judgments themselves anonymous
- Appoint a neutral observer
Net promoter, of course, appears to break one point of that advice, it’s just one big judgment — in the classic question formulation, how likely are you to recommend this product or service to a friend or colleague? But the context of NPS is highly independent. There’s no researcher shepherding a random sample of airport bathroom users into an adjacent interview room to discuss together their experience with the restroom! Your most recent purchase from an online furniture store probably wasn’t made with a large group of friends. You alone respond (or not) to the feedback survey email, and the company benefits from the large group of customers who respond.
But the richness of any survey comes from the context of the answer, not the answer itself. The process of adding context to a simple survey can come from the customer, the transaction, or the environment, and text answers can be categorized. If you bought a chair online and responded to an NPS email survey, the chair itself is context. So is when you bought it, how you paid, what discounts you used, what your first marketing channel was, and so on. When I taught a course called Metrics and Outcomes at the CUNY Newmark Graduate School of Journalism this past spring, I gave my class a mid-term NPS survey. I made the survey truly anonymous to encourage candor so I couldn't cross-tabulate student satisfaction with grades, but the categorization of the textual analysis gave me a really important insight: My class handouts are most appreciated by the students that give me the lowest net promoter scores! In other words, they are my last line of defense, and I can leverage that insight to do a better job teaching the class next semester.
At Alley, we’re constantly making group judgments on our Scrum teams by rating the level of effort of a given task (in Scrum terms, a user story). Scrum’s estimation system beautifully follows nearly all of the advice from Noise: We break down our tasks until they’re small enough to estimate, we vote first with a button click, and the team’s Scrum Master acts as the decision observer. Anonymity is less important here, but most story points are assigned without any concern for who voted what number of points.
When we design stakeholder or audience research, we follow these heuristics as best we can across a wide variety of judgments. When we want to know what audiences think, we’re almost always working with a relatively large number of survey responses and then unpacking a subset of those responses in qualitative interviews. The most salient kind of research, however, often comes from smaller groups of direct decision-makers, each of whom brings their own set of perspectives and responsibilities into a project.
One really neat way to present structured subjudgements in the survey stage also lets us apply factorial analysis, which is a sneaky way to ask quantitative questions without making your survey respondents do math. Presenting various scenarios that combine various factors and then asking the respondent to rate each scenario subjectively lets us use software to assign values to individual features.
We’re testing three different variables at once: Whether or not we launch with a mobile app, whether or not we increase the editorial team size, and a parameter for a hard member wall. By asking several scenarios with different combinations, we can isolate the underlying variables without directly asking the respondents to rate them.
Doing research among stakeholders is a great way to help project leadership see where to steer an initiative, but we have to be subtle about it — nobody wants to take a two-hour test at work! But we need not do that. Any ideation exercise with post-its or in FigJam can be structured as a group judgment just by putting individual work first, breaking down the problem, appointing a neutral observer (usually a member of the Alley strategy team), and ideally, making the work at least superficially anonymous.
