How much do you think that dead ox weighs?

(Verbatim transcript from a fascinating story on NPR this morning, about "the wisdom of the crowd" and The Good Judgment Project, which seeks to apply the wisdom of crowds to predict geopolitical events.)

All of his studies brought [Philip] Tetlock to at least two important conclusions.

First, if you want people to get better at making predictions, you need to keep score of how accurate their predictions turn out to be, so they have concrete feedback.

But also, if you take a large crowd of different people with access to different information and pool their predictions, you will be in much better shape than if you rely on a single very smart person, or even a small group of very smart people.

"The wisdom of crowds is a very important part of this project, and it's an important driver of accuracy," Tetlock said.

The wisdom of crowds is a concept first discovered by the British statistician Francis Galton in 1906.

Galton was at a fair where about 800 people had tried to guess the weight of a dead ox in a competition. After the prize was awarded, Galton collected all the guesses so he could figure out how far off the mark the average guess was.

It turned out that most of the guesses were really bad — way too high or way too low. But when Galton averaged them together, he was shocked:

The dead ox weighed 1,198 pounds. The crowd's average: 1,197.

Finding The True Signal

"There's a lot of noise, a lot of statistical random variation," Tetlock said. "But it's random variation around a signal, a true signal, and when you add all of the random variation on each side of the true signal together, you get closer to the true signal."

In other words, there are errors on every side of the mark, but there is a truth at the center that people are responding to, and if you average a large number of predictions together, the errors will end up canceling each other out, and you are left with a more accurate guess.

That is the wisdom of the crowd.

The point of the Good Judgment Project was to figure out if what was true for the dead ox is true for world events as well.

It is.