A Common Analytic Fail is a freemium blogging platform. On the company blog, Ghost founder and lead designer John O’Nolan wrote a post talking about how the company was using analytics to improve the user experience and get more paid users: How we Figured Out What Makes People Love Ghost 1,000% More

Except they didn’t yet, really.

The data showed that there was a strong correlation between users who created custom themes and those that subscribed to a pro version (see the graphic above).

So the team put together tools that helped new users choose and implement custom themes. Users who didn’t choose a custom theme were prompted with a short video showing them how to do it — and this resulted in many more people implementing a custom theme.

The post then asserts that:

“Quadruple the users entering this point of the conversion funnel, means a trickle-down effect as users with custom themes continue to convert better at the end of the funnel.”

But it’s clear from the rest of the post and the comments that they hadn’t actually tested this. The post says that:

“We don’t know the order of causation at this point…  People who are getting value from Ghost have a custom theme, and people who have a custom theme are getting value from Ghost.”

“This is important.”

Well, no, not necessarily. It’s clear (to me at least) that there’s a good chance that the causation is operating in the opposite direction: that it’s not that choosing a theme makes you more likely to blog more, it’s that more dedicated bloggers, who are more likely to want the premium option, also want their own customized theme.

When a commenter points this out, John says:

“…people who are engaged and people who have a theme are correlated. So by increasing one, we will increase the other.” [my emphasis]

Well, no. If the causation is the other direction, getting non-dedicated bloggers to add custom themes could have zero effect.

John does seem to understand this:

“The theoretical logic here is that by helping people set up a theme, they will subsequently be more engaged :)It’s all very much a work in progress experiment, however – and we’re learning a tremendous amount along the way.”

But we’re left with a huge gap between the data they’ve found so far and the claimed impact. The assertion that they “figured out how to make people love Ghost 1,000x more” seems an exaggeration at many different levels.

Why did I pick on this post? Because it illustrates in a small way what I often see in organizations: some data is collected, some analysis is done, and conclusions are blithely drawn that simply can’t be inferred from the data. In many ways it’s worse than just guessing, because the participants may set off in the wrong direction with excessive confidence

Study after study shows that it’s not enough to provide data — you also need to be sure that data is being correctly analyzed, and it turns out that analytic skills are typically weak at every level of the organization. If you really want to improve analytics, spend less time on the technology, and more time on making sure people know how to use it correctly!

Did I get something wrong in my analysis of Ghost’s post? Please tell me — that’s what analysis is all about!






3 responses to “A Common Analytic Fail”

  1. Christine Avatar

    I agree with you that Ghost has taken a narrow view of their data. I commonly alert my clients that this can be hazardous and cause them to make changes to their sites that actually hurt their conversion.

    Are they looking at the frequency of posts from unsubscribed vs. subscribed? Traffic for each? Is there some other demographic that accounts for it? You need to ask more than one question to get the answer you need.

  2. Pat Hennel Avatar

    “In many ways it’s worse than just guessing, because the participants may set off in the wrong direction with excessive confidence”

    That’s the danger that comes with data analysis. If you don’t have the whole picture and account for everything chances are you missing the “real” point.

  3. martin english (@martin_english) Avatar

    There are even more basic errors than this;
    A recent government report in Australia stated that that the mean broadband bandwidth used by Australian families in 2020 would 15MBPS. All the mainstream news reporting (Free-to-air and pay TV news, Newspapers, etc) reported it as the average family would use 15MBPS