{"id":12450,"date":"2015-02-10T17:34:22","date_gmt":"2015-02-10T16:34:22","guid":{"rendered":"http:\/\/timoelliott.com\/blog\/?p=6966"},"modified":"2015-02-10T17:34:22","modified_gmt":"2015-02-10T16:34:22","slug":"a-common-analytic-fail","status":"publish","type":"post","link":"https:\/\/timoelliott.com\/blog\/2015\/02\/a-common-analytic-fail.html","title":{"rendered":"A Common Analytic Fail"},"content":{"rendered":"

\"ghost.org\"Ghost.org is a freemium<\/a> blogging platform. On the company blog,\u00a0Ghost founder and lead designer\u00a0John O\u2019Nolan<\/a>\u00a0wrote a post talking about how the company was using analytics to improve the user experience and get more paid users:\u00a0How we Figured Out What Makes People Love Ghost 1,000% More<\/a><\/p>\n

Except they didn’t yet, really.<\/p>\n

The data showed that there was a strong correlation<\/em> between users who created custom themes and those that subscribed to a pro version (see the graphic above).<\/p>\n

So the team put together tools that helped new users choose and implement custom themes.\u00a0Users who didn’t choose a custom theme were prompted with a short video showing them how to do it — and this resulted in many more people implementing a custom theme.<\/p>\n

The post then asserts\u00a0that:<\/p>\n

“Quadruple the users entering this point of the conversion funnel, means a trickle-down effect as users with custom themes continue to convert better at the end of the funnel.”<\/p><\/blockquote>\n

But it’s clear from\u00a0the rest of the post\u00a0and the comments that they hadn’t\u00a0actually tested\u00a0this.\u00a0The post says that:<\/p>\n

“We don\u2019t know the order of causation at this point… \u00a0People who are getting value from Ghost have a custom theme, and people who have a custom theme are getting value from Ghost.”<\/p><\/blockquote>\n

“This is important.”<\/em><\/p><\/blockquote>\n

Well, no, not necessarily.\u00a0It’s clear (to me at least) that there’s a good chance that the causation is operating in the opposite direction: that it’s not\u00a0that\u00a0choosing a theme makes you more likely to blog more,\u00a0it’s that more dedicated bloggers, who are more likely to want the premium option, also want their own customized theme.<\/p>\n

When a commenter points this out, John says:<\/p>\n

“…people who are engaged and people who have a theme are correlated. So by increasing one, we will increase the other.<\/strong>” [my emphasis]<\/p><\/blockquote>\n

Well, no. If the causation is the other direction, getting non-dedicated bloggers to add custom themes could have zero effect.<\/p>\n

John does seem to understand this:<\/p>\n

“The theoretical logic here is that by helping people set up a theme, they will subsequently be more engaged :)It’s all very much a work in progress experiment, however – and we’re learning a tremendous amount along the way.”<\/p><\/blockquote>\n

But we’re left with a huge gap between the data they’ve found so far and the claimed impact. The assertion that they\u00a0“figured out how to\u00a0make people love Ghost 1,000x more” seems\u00a0an exaggeration at many different levels.<\/p>\n

Why did I pick on this post? Because it illustrates in a small way what I often see in organizations: some data is collected, some analysis is done, and conclusions are blithely drawn that simply can’t be inferred\u00a0from the data.\u00a0In many\u00a0ways it’s worse than just guessing, because the participants may\u00a0set off in the wrong direction\u00a0with excessive confidence<\/p>\n

Study after study shows that it’s not enough to provide data — you also need to be sure that data is being correctly analyzed, and it turns out that analytic skills are typically weak at every level of the organization. If you really want to improve analytics, spend less time on the technology, and more time on making sure people know how to use it correctly<\/a>!<\/p>\n

Did I get something wrong in my analysis of Ghost’s post? Please tell me — that’s what analysis is all about!<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

If you’re going to tout the benefits of your analysis, back it up with data!<\/p>\n","protected":false},"author":2,"featured_media":12522,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[14],"tags":[96,100,158,204,230,295],"class_list":["post-12450","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-thoughts","tag-analysis","tag-analytics","tag-best-practice","tag-business-intelligence","tag-causation","tag-correlation"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2015\/02\/ghost.org_-608x267.png?fit=608%2C267&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p3X9RF-3eO","_links":{"self":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts\/12450","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/comments?post=12450"}],"version-history":[{"count":0,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts\/12450\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/media\/12522"}],"wp:attachment":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/media?parent=12450"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/categories?post=12450"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/tags?post=12450"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}