November 24, 2009

Not far from Lake Wobegon (1), there is a town where nobody is of average height. Contrary to its more stellar neighbor, this is simply because its population is divided between blue dwarves and red giants. Progress reaches it unevenly. To this day the town library is its only source of movies.

Its digital isolation had made it an attractive testbed for Netslips, which models consumers' movie preferences. Results from its study were as striking as its statistics were robust. The reds like action movies and the blues light comedies. As Netslips was announcing its new, on demand, online movie rental system, someone attacked the study in a letter to the local newspaper as having no more substance than the Emperor's clothes.

The critic even boasted he could change the preferences of his fellow citizens overnight. While her advisers hotly disputed his credentials and refused to debase science to his level, Netslips' CEO signed a contract with him for sharing how he would do it. Elementary, he said after having cashed her check, I would switch top and bottom shelves at the library, where currently action movies are on the top and light comedies on the bottom.

"In order to formulate its guidelines, the task force used new data from mammography studies in England and Sweden and also commissioned six groups to make statistical models to analyze the aggregate data." writes Gina Kolata apropos the recent recommendation to drop breast cancer screening for women below 50 (*). Is the previous apologue an attempt to discredit this task force and its startling conclusion?

Not at all. But neither can we ignore the current brouhaha. It tells us something about the nature of truth without which our Information Age may well turn out to be another Dark Age. The stakes are high and personal. As Sheila M. Rothman declared to Kevin Sack (**) "the public state of mind right now is that they're frightened that evidence-based medicine is going to be equated with rationing. They don't see it in a scientific perspective."

Perhaps we should step back from healthcare and go to court for instance. There, statistical evidence is as big an issue, witness Amit Pundik, a legal scholar, who illustrates it with the "gate-crasher paradox" (***). Two persons are accused of this misdemeanor. One because of an eyewitness known to be right 9 times out of 10, the other because he was in a crowd of 1000 spectators who had among themselves bought 100 tickets. On this sole evidence, can there be a valid reason to treat the accused differently, nailing the former and letting the latter free? No, he answers.

Amit Pundik himself stresses that what he calls "an epistemic principle" should "apply regardless of whether the agent is a juror, scientist, or layman". Implicitly accusing the courts of hiding their ignorance of sound principles behind the nitty-gritty of each case, isn't he in league indeed with Brent James, the physician featured in David Leonhardt's article on evidence-based care (****)? "Human beings can [...] be unduly influenced by just a few experiences. {...} As a result, different doctors frequently end up coming up with different answers to the same question."

Religion is not served by scientific claims made on its behalf. Just as dangerous is the over extension of scientific truth by its well-meaning advocates. The Netslips case does not belittle statistical evidence. It warns us not to label those who resist it as intellectually challenged. Indeed, much of the brouhaha comes from a spurious confusion between science and authority, which, together with popularity, are the three sources of Truth.

Statistics of course belong to science and anyone can convince oneself by its demonstrations as by any mathematical theorem. But its self-evidence goes no further than some event space (2) and its associated probability measure (3). It does not extend to the reality of one event (4). On the other hand, decisions about what happened or need to be done must be taken all the time in the absence of proof. If, in order to do so, we choose to believe a person who claims to be right, we receive the truth this person witnesses on the basis of authority, not from science.

In this context, statistical evidence plays the role of an orphan put out for adoption. Legitimate child of science, it shapes the conviction of those who believe the underlying model truly reflects the real world (5), often on the authority of someone else if not from sheer popularity (6). Why then be surprised to see conflicts between independent authorities and their followers? It's perfectly normal. Or to see science falsely wielded as a weapon to wound the opposing party? It's only too human. Would religion never be used, whenever handy, for political gains!

Hasn't the gate-crasher paradox suddenly lost its mystery? If the eyewitness is taken as an authority, the two cases are utterly dissimilar. If on the contrary authority is declined by the eyewitness or rejected by the jury, the two cases are indeed similar. Either both defendents are acquitted on benefit of doubt or condemned, if for instance they can rely on a fair compensation in case they later clear themselves by producing a valid ticket.

There is no escaping the fact that statistical evidence is the step child of authority. Suppose a jury rejects an eyewitness' authority. Once reached on the only basis left available, i.e. statistical evidence, the jury's decision whatever it is will become truth for society on the very jury's authority.

Going back to healthcare, we can better appreciate David Leonhardt's fine relation, quoting Robert Wachter, "an expert on medical errors". "[Dr. James] knows that the minute he says, 'I'm right, and you must do this,' he loses everybody but the true believers". That is indeed how authority works best, convincing from the inside rather than through the exercise of power. What else justifies the physicians' authority? "It is the faith that patients have in their doctors" says William Lewis, a productivity expert.

Important though they are, conflicts of interest may well be less of an issue in healthcare as conflicts of independent authorities. The patients, their physicians, the medical teams which write textbooks and protocols, the administrators in charge of socialized budgets, whether public or private. Administrators limit budgets, boards certify physicians, physicians sign prescriptions, patients choose whether to comply. What should be done?

Since authorities realistically resort to raw power, the first remedy is to streamline conflicts between them. A tall order in a pronaocracy which calls itself a democracy. If expenses come above budget, shareholders or electors should fire the administrators. But who will dare speak about rationing when it is easier to say science will solve it all and lay the blame at the door of the weakest authorities, the physicians and the poorest patients?

Second one should nurture financial independence and humility as the right conditions to pursue statistical evidence. Data can be forged, models can go wrong, circumstances can change. Any result paid for by an interested party will therefore be suspect, but then where to find the money? A tall order too. And neither humility nor financial independence will prevent vilification by those who oppose the results, witness the mammography study.

Last, one should not forget a scientific etiology, when found, is worth a hundred statistics. Rob Walker recently wrote about "an Internet radio service called Pandora" (*****). Ironically its fans appear interested less in its science than its authoritative recommendations. Unfazed, it pays "musicologists" to eliminate the influence of popularity and express both tastes and songs in a common language to find matches they can explain.

But if you cannot prove the truth, do not unburden your own responsibility on science, take a calculated gamble and bear witness with authority!

Philippe Coueignoux

• (*) ......... New Guidelines Suggest Fewer Mammograms, by Gina Kolata (New York Times) - November 17, 2009
• (**) ....... A Medical Culture Clash, by Kevin Sack (New York Times) - November 20, 2009
• (***) ..... Could there be any Epistemic Reason to Restrict the Use of Statistical Evidence in Court?, by Amit Pundik (University of Cambridge) - May 2008
• (****) ... Dr James Will Make It Better, by David Leonhardt (New York Times Magazine) - November 8, 2009
• (*****) . The Song Decoders, by Rob Walker (New York Times Magazine) - October 18, 2009
• (1) for this place where "all children are above average", see Lake Wobegon in the wikipedia
• (2) for a brief definition, see event space in the French wikipedia
• (3) for a scientific definition, see measure in the wikipedia
• (4) for further explanations, see probabilité in the French wikipedia
• (5) Science also models reality. But scientific models are open to falsification by singular contrarian events, statistical models are not.
• (6) according to the last wikipedia entry, popularity was how no less of an authority than Aristotle himself defined probability. On en perd son latin.