Remember also Gorman's experiments in the 1980s testing methodologies for solving New Eleusis puzzles. The winning strategy there was "confirm early, disconfirm late".

From memory these were small, but the result fits into a much larger and more robust literature on explore/exploit scenarios in foraging. Explore early and switch to exploit based on expected value. (See also Thompson sampling and multi-armed bandits.)

Indeed in aggregate these 7 also do not do so.

Again, possibly because Heuer's work has already mitigated confirmation bias in the intelligence community.

Specific citation of good studies on practicing analysts revealing lack of confirmation bias.

If so we might credit the ubiquity of Heuer's work, as well as general knowledge of confirmation bias in the decades preceding the studies.

Nice example of a specific, common, & documented bias that ACH might inadvertently reinforce.

Lack of definition is surprising but maybe not fatal if people intuitively get to the same place and it's nearly normative.

Assuming ACH is meant to disregard weak evidence that's about equally relevant to all theories, then inconsistent should mean likelihood ratio <<1, and consistent mean likelihood ratio >>1.

While few would use likelihoods, that's roughly: * Data expected more on other theories than this one * Data expected more on this theory than on others

So this invitation to noise might have worked out OK.

The rule to ignore confirming evidence may be intended to thwart a tendency to pick evidence that supports my theory, without realizing it supports alternatives about as well. Certainly we see this all the time informally.

But if that tendency affects analysts, it seems not to show up in ACH experiments.

7 studies, 6 participant samples: * Lehner: 12 experienced [US?] analysts (of 24 total) * Mandel: 50 UK analysts * [Dhami: same 50 analysts] * Karvetski1: 147 Canadian citizens * Karvetski2: 227 Canadian citizens * Whitesmith: 7 UK intel + 39 KCL staff/stus * Maegherman: 191 Dutch & Belgian Law Faculty

So of the 6 participant samples, 3 had intel analysts (N=12, 50, 7) and 1 had 191 Law Faculty evaluating a legal case, so very much trained and very much relevant.

Not a meta-analysis.

If there's a fatal selection bias, this seems the likeliest place. For example, if ACH > X, and X > none, it's likely ACH > none.

It seems unlikely we have evidence of X > none for many other SATs, but even X performing similar to none would let ACH vs X be a rough proxy for ACH vs none.

And some related (but not necessarily SAT) areas where I think we know X > none: * Frequency-formats for probability calculations * Brainstorming separately, then combining (for ideation) * Peer instruction (for learning) * Argument maps (for critical thinking) * Aggregation (for estimation & forecasting) * Specific weighting techniques for better aggregation * Probability training (for estimation & forecasting) * Selecting superforecasters ( " ) * Tracking performance ( " ) (+ probably most things)

Main contribution of this paper: update synthesis.

This is key. In the pervasive explore/exploit tradeoffs, insufficient exploration (confirmation) can prevent you from finding the best hypothesis/explanation/resource.

Remember Mike Gorman's (1980s?) attempts to model science with the New Eleusis card game. Given an initial set of cards laid out according to a rule, participants had to guess the rule by playing cards to test it. Small studies, but consistent with explore/exploit tradeoffs, "confirm-early-disconfirm-late" teams solved the rules faster than either "disconfirm" or "confirm" teams.

Yes. But many scientists resonate with the idea of falsification. When theories multiply faster than data, a sharp razor beckons, even if we don't know how to wield it.