![\"esp_banner\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/01\/esp_banner.jpg?resize=690%2C310&ssl=1\" "\\"esp_banner\\"")
<\/p>\n<\/p>\n
The New York Times reports<\/a> that a respected psychology journal is due to publish a paper purporting to show \u201cstrong evidence\u201d for extra-sensory perception:<\/p>\n \u201cA software program randomly posted a picture behind one curtain or the other \u2014 but only after the participant made a choice. Still, the participants beat chance, by 53 percent to 50 percent, at least when the photos being posted were erotic ones. They did not do better than chance on negative or neutral photos.\u201d<\/p><\/blockquote>\n Crucially, no \u201ctopflight statisticians\u201d were part of the peer review. When I was at university, struggling to use a sophisticated statistics package on a mainframe as part of my econometrics degree, I dreamed of having a program that would just cruise through all the possible combinations of variables, and tell me which ones were correlated. That ability now exists, but the danger is that few people realize how much higher the bar must be set for a result to be deemed significant in such circumstances.<\/p>\n Given a large enough set of random numbers, you will always be able to find a \u201csignificant\u201d relationship \u2013 especially if that\u2019s exactly what you\u2019re looking and hoping for.<\/p>\n To me, the experiment above sounds like it may have this problem \u2013 for example, if there were lots of different categories of photos, and the \u201csignificant\u201d relationship was cherry-picked from the available results. And even if the level of significance has indeed been increased to take account of this, the result could still be random (if there\u2019s a choice between changing everything we know about science and it being a fluke result, I\u2019m going with the latter).<\/p>\n In science, thankfully, it\u2019s easy for somebody else to repeat the experiment and validate the correlation, ideally before a respected journal makes a fool of itself (although I suspect they\u2019re simply making a calculated bid for more publicity, and it\u2019s working very successfully).<\/p>\n In business, it\u2019s much harder to know if your \u201cresults\u201d are valid. The same problem exists \u2013 people are looking for a certain type of result, and keep running the numbers until they find something that looks like a relationship: \u201cLook! Customer satisfaction is correlated with their age!\u201d . But it\u2019s much harder to \u201crerun the experiment\u201d, and businesses don\u2019t always have\/take the time to check their results.<\/p>\n Despite having worked in BI for over twenty years (or maybe because of it), I\u2019m deeply distrustful of most corporate analytics. I believe business analytics is essential, but that it\u2019s also essential to assume that any relationship you find is a working hypothesis, to be validated through further analysis (e.g. correlation is not causation<\/a>), and expert discussion (as with peer-reviewed science papers, the best way to deal with potential analysis problems is greater transparency — social BI technologies like Streamwork<\/a> are becoming increasingly important).<\/p>\n