Relying on Data Can Lead to the Wrong Decisions Says


I just read a new CFO Magazine article called “That New Big Data Magic”, which contains the following opening paragraphs:

Back in the 1980s, American Airlines (AA) was partnered with British Airways (BA), and AA’s marketing head wanted to know how many of the airline’s gold card members, its most profitable customers, were flying BA rather than AA. Larry Tieman, then a managing director in IT at AA, dove into the massive amounts of data AA had collected on its customers and reported back that, basically, all of AA’s gold card holders were flying BA. The reason seemed obvious: BA’s service simply was better. So AA launched a huge quality-improvement initiative. It upgraded its food service and changed its flight schedules to increase on-time rates. It invested heavily to compete on quality on all fronts.

The results, Tieman recalls, were disastrous.

“None of it mattered,” says Tieman, who between 2000 and 2010 was a senior technology vice president at FedEx. “The quality of food meant nothing. What was sticky was frequent flyer loyalty programs.”

“With Big Data,” Tieman says, “you may be spot-on about a problem, but the solution doesn’t magically appear out of the data.”

The story goes on to say:

“Relying on data alone could lead a company down the path AA took: investing heavily in the wrong things.”

I re-read this several times, and still couldn’t make any sense out of the anecdote:  if frequent flyer loyalty was key, why didn’t the AA fliers stick with AA flights to get the miles? I assume the partnership gave them air miles when they used BA – but presumably they could still get miles on AA, too, so the part about “what was sticky was the frequent flier program” is true but irrelevant and we’re back to the original question, which remains unanswered: why did they fly BA rather than AA?

The anecdote doesn’t back up the conclusion, leading to lots of other questions:

  • Did flying BA give them more miles than BA through the partnership? (sounds like a strange partnership agreement)
  • Were the people flying BA only there was no AA flight available? (marketing head wanted to know when people were flying BA rather than AA, so this would be bad analysis)
  • Were the BA flights free ones, using up the air miles? (even so, what difference would it make to the choice equation?)
  • Did the gold card fliers also have BA cards, and these trumped their AA cards? (I know they used to offer gold cards to people who already had gold cards on other airlines, with the logic that these were good customers, that they wouldn’t change carriers without similar conditions, and the marginal cost of extra people in the priority lineup, etc. is very low)
  • Was the routing different/better (this would explain why food service and schedules didn’t make any difference, but would change the conclusion of what was important)…

In order to find out what was going on, I swapped emails with Larry Tieman, whose quote in the article explains why it’s wrong to “rely on data alone” (you need people):

“what you do with [data]is a people-based activity, a skill base you have to mature. And it doesn’t come quickly.”

It turns out that frequent fliers did indeed get air miles on either airline, and that the “obvious reason” was also the real one: on head-to-head flights, frequent fliers did indeed choose BA because of “the quality of business and first class”. The “quality from the ground up” initiative mentioned cost $1Bn, with little impact: as the last two decades has shown, what most fliers are interested in is lower prices.

So, unlike the article’s assertion,  AA’s problems don’t seem to have had anything to do with “relying on data alone”. Larry’s conclusion over email was rather different:

“My moral, the data can tell you what is wrong but fixing it is another thing all together.”

Overall, it’s maybe a good example of what plagues most business analysis — the data is incomplete and confusing, and isn’t necessarily related to the most important problems people have to deal with! The rest of the article is less controversial, and include a nice summary of what CFOs (and everybody else) should be thinking about in analytics:

“…new technologies are making it more economical to make sense of Big Data which, in fact, has been around for a long, long time. The caveat is that those technologies will not provide those opportunities. That’s still up to the people who make business decisions.”

Finally, as a footnote, I’m with Andy Bitterer that there are four ‘V’s to the information challenge – not just Volume, Velocity, and Variety, but also Validity…


7 comments to “Relying on Data Can Lead to the Wrong Decisions Says”
  1. Customer surveys could of served as a valuable added data point for decision makers in this case. Granted, this still relies on the right questions being asked and interpretation of of both sets of data by people.

  2. Pingback: Links for September 11, 2011 | Eric D. Brown

  3. Pingback: Field Notes: Thought and Action, Binary Thinking, Business Analytics, Listening and Attention | Marcus Bertrand

  4. Timo,

    I agree with you that they came up with the right conclusion for the analysis – however, the fact that they spent $1B does not mean they achieved parity with BA anyway. Even parity is unlikely to have been enough to unstick customers, they needed to be way above BA quality to overcome any inertia.

    Another aspect is that BA were certainly operating from a much higher “investment” base in terms of quality, so given my experience with American carriers, they probably needed to spend way more than $1B anyway.

    Assuming the amount of spend achieves your goals is not wise. How did they measure the spend was effective and what the perception of their new quality levels were against BA.

    This is all assuming that BA stood still as well. For all we know from this data, maybe BA spend $2B at the same time!


  5. Pingback: Twitted by CapgeminiINBI

  6. This is a truly bizarre line of reasoning. When they say :-

    “Relying on data alone could lead a company down the path AA took: investing heavily in the wrong things.”

    I wonder what else they could rely on, only two spring things to mind

    1) Experience, which I would argue is just an accumulation of data
    2) Guesswork

    whichever it was it failed them in this case.

    The failure here was to stop at the easy to collect data (counting number of people in a program) and not get to the hard data (data on what motivated their customers).

    • Actually, I think what actually happened was that they did the analysis, came to the right conclusion for that analysis (that their premium customers preferred BA because of the service in first-class and business), but completely missed the bigger picture: that the whole market was changing — notably with much more price-pressure for cattle-class, that also came to redefine what companies were willing to pay for business flights…

Comments are closed.