ESP and Business Analytics

esp_banner

The New York Times reports that a respected psychology journal is due to publish a paper purporting to show “strong evidence” for extra-sensory perception:

“A software program randomly posted a picture behind one curtain or the other — but only after the participant made a choice. Still, the participants beat chance, by 53 percent to 50 percent, at least when the photos being posted were erotic ones. They did not do better than chance on negative or neutral photos.”

Crucially, no “topflight statisticians” were part of the peer review. When I was at university, struggling to use a sophisticated statistics package on a mainframe as part of my econometrics degree, I dreamed of having a program that would just cruise through all the possible combinations of variables, and tell me which ones were correlated. That ability now exists, but the danger is that few people realize how much higher the bar must be set for a result to be deemed significant in such circumstances.

Given a large enough set of random numbers, you will always be able to find a “significant” relationship – especially if that’s exactly what you’re looking and hoping for.

To me, the experiment above sounds like it may have this problem – for example, if there were lots of different categories of photos, and the “significant” relationship was cherry-picked from the available results. And even if the level of significance has indeed been increased to take account of this, the result could still be random (if there’s a choice between changing everything we know about science and it being a fluke result, I’m going with the latter).

In science, thankfully, it’s easy for somebody else to repeat the experiment and validate the correlation, ideally before a respected journal makes a fool of itself (although I suspect they’re simply making a calculated bid for more publicity, and it’s working very successfully).

In business, it’s much harder to know if your “results” are valid. The same problem exists – people are looking for a certain type of result, and keep running the numbers until they find something that looks like a relationship: “Look! Customer satisfaction is correlated with their age!” . But it’s much harder to “rerun the experiment”, and businesses don’t always have/take the time to check their results.

Despite having worked in BI for over twenty years (or maybe because of it), I’m deeply distrustful of most corporate analytics. I believe business analytics is essential, but that it’s also essential to assume that any relationship you find is a working hypothesis, to be validated through further analysis (e.g. correlation is not causation), and expert discussion (as with peer-reviewed science papers, the best way to deal with potential analysis problems is greater transparency — social BI technologies like Streamwork are becoming increasingly important).

[Update: there’s a great New Yorker Article that talks about the issues of finding, and replicating, significant results in studies — much of accepted science may be false?]

Comments

5 responses to “ESP and Business Analytics”

  1. Al Avatar

    Not sure if you need to be a top flight statistician to spot the flaws – is beating chance 53 percent to 50 statistically significant? Well it depends on sample size. Obviously if this test was only run a few times then this difference could easily be by chance itself, if it was run say 10,000 times then it would be interesting. Article doesn’t state how often this test was run so we don’t know.

    See this handy statistical significance calculator to work out how big the trial size should be for this result to be statistically significant – http://www.prconline.com/education/tools/statsignificance/index.asp

    @natasha – drug trials have massive financial pressure to show a positive result. Throw in media bias (they love their miracle cures) and it’s not surprising that drug testing is flawed. I think this is more a problem with humans rather than the scientific method

    1. Timo Elliott Avatar

      Al,

      Since the article was due to be published in a reputable journal, I was assuming that the most basic hurdle for statistical significance had been jumped — thanks for the link, though!

  2. iip albanjary Avatar

    so,
    Business is like an art, art doesn’t need exact science

  3. Natasha Lloyd Avatar
    Natasha Lloyd

    Apparently, it’s not that easy to repeat such experiments even in science: http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all. Scary to think what the decline effect could mean for BI.

    1. Timo Elliott Avatar

      Great article — thanks!