A Common Analytic Fail

Ghost.org is a freemium blogging platform. On the company blog, Ghost founder and lead designer John O’Nolan wrote a post talking about how the company was using analytics to improve the user experience and get more paid users: How we Figured Out What Makes People Love Ghost 1,000% More

Except they didn’t yet, really.

The data showed that there was a strong correlation between users who created custom themes and those that subscribed to a pro version (see the graphic above).

So the team put together tools that helped new users choose and implement custom themes. Users who didn’t choose a custom theme were prompted with a short video showing them how to do it — and this resulted in many more people implementing a custom theme.

The post then asserts that:

“Quadruple the users entering this point of the conversion funnel, means a trickle-down effect as users with custom themes continue to convert better at the end of the funnel.”

But it’s clear from the rest of the post and the comments that they hadn’t actually tested this. The post says that:

“We don’t know the order of causation at this point…  People who are getting value from Ghost have a custom theme, and people who have a custom theme are getting value from Ghost.”

“This is important.”

Well, no, not necessarily. It’s clear (to me at least) that there’s a good chance that the causation is operating in the opposite direction: that it’s not that choosing a theme makes you more likely to blog more, it’s that more dedicated bloggers, who are more likely to want the premium option, also want their own customized theme.

When a commenter points this out, John says:

“…people who are engaged and people who have a theme are correlated. So by increasing one, we will increase the other.” [my emphasis]

Well, no. If the causation is the other direction, getting non-dedicated bloggers to add custom themes could have zero effect.

John does seem to understand this:

“The theoretical logic here is that by helping people set up a theme, they will subsequently be more engaged :)It’s all very much a work in progress experiment, however – and we’re learning a tremendous amount along the way.”

But we’re left with a huge gap between the data they’ve found so far and the claimed impact. The assertion that they “figured out how to make people love Ghost 1,000x more” seems an exaggeration at many different levels.

Why did I pick on this post? Because it illustrates in a small way what I often see in organizations: some data is collected, some analysis is done, and conclusions are blithely drawn that simply can’t be inferred from the data. In many ways it’s worse than just guessing, because the participants may set off in the wrong direction with excessive confidence

Study after study shows that it’s not enough to provide data — you also need to be sure that data is being correctly analyzed, and it turns out that analytic skills are typically weak at every level of the organization. If you really want to improve analytics, spend less time on the technology, and more time on making sure people know how to use it correctly!

Did I get something wrong in my analysis of Ghost’s post? Please tell me — that’s what analysis is all about!