Here’s one real-life example of the limits in quantitative thinking observed in an average company, shared with me by Michael Thompson. He recently worked with Dan Rosenberg’s UX team on a SAP research project that isolated over 100 distinct, real-life uses of business analytics across dozens of roles and business functions.

A retail analyst was attempting to figure out the success of product promotions on the “sales” page of a web site. After gathering the data, he used Excel to create the table below. He compared the increase in each product sold with and without the promotions, and then tried to calculate the average percentage increase across the different products. The result was that “the promotions increased sales by 151% on average”.

This result was then used to create a rule-of-thumb of how much extra inventory the company should have on hand at the start of a promotion: “roughly double”. This lowball figure, it was felt, would generally provide enough stock for promotions, while reducing the chances that the company would be left with too much stock at the end of the promotion.

But there are a whole bunch of problems with this analysis. The most basic mathematical problem is that, while there are exceptions, calculating the average of percentages almost always gives an unwanted answer.

A better way to calculate the success of the promotions is to compare the average sales of all the products before and after a promotion (see table below). This shows that the promotions increased the average quantity of products sold by only 71% – considerably less than double.

* note that it would be it would be easier to compare the % increase in totals items sold, which would come out to the same result, but I’ve used the averages to make the comparison between the two methods as obvious as possible.

So if the company did use the proposed rule of thumb they might have a nasty shock. Instead of an expected “lowball” inventory that would be used up by the promotions, they should instead expect to have an extra 29% of the original items left over at the end.

And beyond the numbers, there’s a lot of unanswered questions about the analysis that call into question its usefulness:

• Is units sold the best measure? Wouldn’t it be better to work with revenue or profit?
• There’s huge variability between the different categories of product. Is the notion of an average even meaningful? Maybe categories of products could be created for more meaningful analysis?
• Do these percentages stay reliable over successive promotions, or are they also very variable?
• Is it in fact better to have too little stock than too much? By how much?
• Might external variables or seasonal variations have lead to the observed results, rather than the promotion? (e.g. maybe it was the 4th of July, and charcoal sales would have soared anyway)

In general, people are tempted to throw up their hands at this point and say “well, we have to make some sort of prediction for inventory levels – isn’t this better than nothing?” And the answer is yes (as long as you get the basic math right!), but somebody in the company should be aware of how limited the analysis currently is, and be constantly trying to improve it. Unfortunately, as in this case, real-world business people often rely on basic, faulty analysis, and lack the curiosity (or incentive?) to push it any further.

Given human nature, it’s probable that this example is not an isolated case. If you really want to improve your company’s return on investment in analytics, you may want to consider investing in more training rather than yet another technology solution.

I just read a new CFO Magazine article called “That New Big Data Magic”, which contains the following opening paragraphs:

Back in the 1980s, American Airlines (AA) was partnered with British Airways (BA), and AA’s marketing head wanted to know how many of the airline’s gold card members, its most profitable customers, were flying BA rather than AA. Larry Tieman, then a managing director in IT at AA, dove into the massive amounts of data AA had collected on its customers and reported back that, basically, all of AA’s gold card holders were flying BA. The reason seemed obvious: BA’s service simply was better. So AA launched a huge quality-improvement initiative. It upgraded its food service and changed its flight schedules to increase on-time rates. It invested heavily to compete on quality on all fronts.

The results, Tieman recalls, were disastrous.

“None of it mattered,” says Tieman, who between 2000 and 2010 was a senior technology vice president at FedEx. “The quality of food meant nothing. What was sticky was frequent flyer loyalty programs.”

“With Big Data,” Tieman says, “you may be spot-on about a problem, but the solution doesn’t magically appear out of the data.”

The story goes on to say:

“Relying on data alone could lead a company down the path AA took: investing heavily in the wrong things.”

I re-read this several times, and still couldn’t make any sense out of the anecdote:  if frequent flyer loyalty was key, why didn’t the AA fliers stick with AA flights to get the miles? I assume the partnership gave them air miles when they used BA – but presumably they could still get miles on AA, too, so the part about “what was sticky was the frequent flier program” is true but irrelevant and we’re back to the original question, which remains unanswered: why did they fly BA rather than AA?

The anecdote doesn’t back up the conclusion, leading to lots of other questions:

• Did flying BA give them more miles than BA through the partnership? (sounds like a strange partnership agreement)
• Were the people flying BA only there was no AA flight available? (marketing head wanted to know when people were flying BA rather than AA, so this would be bad analysis)
• Were the BA flights free ones, using up the air miles? (even so, what difference would it make to the choice equation?)
• Did the gold card fliers also have BA cards, and these trumped their AA cards? (I know they used to offer gold cards to people who already had gold cards on other airlines, with the logic that these were good customers, that they wouldn’t change carriers without similar conditions, and the marginal cost of extra people in the priority lineup, etc. is very low)
• Was the routing different/better (this would explain why food service and schedules didn’t make any difference, but would change the conclusion of what was important)…

In order to find out what was going on, I swapped emails with Larry Tieman, whose quote in the article explains why it’s wrong to “rely on data alone” (you need people):

“what you do with [data]is a people-based activity, a skill base you have to mature. And it doesn’t come quickly.”

It turns out that frequent fliers did indeed get air miles on either airline, and that the “obvious reason” was also the real one: on head-to-head flights, frequent fliers did indeed choose BA because of “the quality of business and first class”. The “quality from the ground up” initiative mentioned cost $1Bn, with little impact: as the last two decades has shown, what most fliers are interested in is lower prices.

So, unlike the article’s assertion,  AA’s problems don’t seem to have had anything to do with “relying on data alone”. Larry’s conclusion over email was rather different:

“My moral, the data can tell you what is wrong but fixing it is another thing all together.”

Overall, it’s maybe a good example of what plagues most business analysis — the data is incomplete and confusing, and isn’t necessarily related to the most important problems people have to deal with! The rest of the article is less controversial, and include a nice summary of what CFOs (and everybody else) should be thinking about in analytics:

“…new technologies are making it more economical to make sense of Big Data which, in fact, has been around for a long, long time. The caveat is that those technologies will not provide those opportunities. That’s still up to the people who make business decisions.”

Finally, as a footnote, I’m with Andy Bitterer that there are four ‘V’s to the information challenge – not just Volume, Velocity, and Variety, but also Validity…

I’ll confess upfront that I’ve always been slightly dubious about the assertion that “sustainability is about making money”. After all, if it’s about making money, it’s something that companies should be doing anyway – it’s only when there’s a tradeoff between sustainability and money that there’s room for interesting debate about to what extent it should be one of the goals of the organization.

However, as Diversey’s CEO Curt Johnson points out:

“CO₂ is waste, so if you minimize CO₂, you minimize waste and you maximize efficiency and increase profits”

It turns out that putting in the effort to calculate the carbon footprint of the entire supply chain, from raw products to final consumption, is a great opportunity to identify savings that might previously have been missed by more step-by-step, siloed efficiency efforts.

Walkers Crisps is a great example of this. The company sells potato chips, or “the UK’s favorite crisps” as they are known locally, and is part of Frito-Lay, which is in turn a division of PepsiCo. In 2007, Walkers became the first company in the world to display a carbon reduction logo on their product, and discovered considerable savings in the process. From The Economist:

“The process of calculating the carbon footprint for Walkers crisps revealed an unexpected opportunity to save energy. It turned out that because Walkers was buying its potatoes by gross weight, farmers were keeping their potatoes in humidified sheds to increase the water content. Walkers then had to fry the sliced potatoes for longer to drive out the extra moisture. By switching to buying potatoes by dry weight, Walkers could reduce frying time by 10% and farmers could avoid the cost of humidification. Both measures saved money and energy and reduced the carbon footprint of the final product.”

The report doesn’t mention what products and procedures were used to do the carbon footprint analysis, but “Business Objects is the key database for Walkers” for reporting, according to the PepsiCo UK web site. You can read more about the carbon footprint of a packet of cheese & onion crisps on their web site.

Viewed in this way, calculating the carbon footprint of your products is “just” another form of efficiency analytics. If you’re interested in doing this for your organization, you can read more about the business case for sustainability, SAP products that can help you calculate your carbon footprint and view SAP’s own Sustainability Report.

Why did it take so long to discover the source of the deadly outbreak of E.coli poisoning in Germany? Since the start of the outbreak, 35 people have died, over 100 will need kidney transplants or life-long dialysis, more than 3,000 people are ill, and falsely-accused Spanish cucumber producers have lost millions of Euros.

Researchers initially relied on the most obvious source of data: the tests carried out on suspect food. Unfortunately, the thousands of tests on farms throughout the country came up negative, even in the organic farm in Lower Saxony that now seems to be at the cause of the problem. They also carried out two surveys. The first showed that people who consumed raw tomatoes, cucumbers, and lettuce were more often sick than the others, and another case study showed a strong association between the consumption of foods from the salad bar and E. coli infection – but this wasn’t enough to narrow down the origin of the outbreak.

It was only when a detailed cohort study was carried out, including one group of nineteen Swedes that had all eaten in the same restaurant, that the data showed that those who had eaten the sprouts were nine times as likely to have gotten sick.

According to Daryll E. Ray & Harwood D. Schaffer:

“To ascertain the consumption of raw fruit and vegetables by patients and controls more objectively and less dependently on memory, RKI used the following approach in the ‘recipe-based restaurant cohort study:’ Five groups (travel groups, clubs, etc.) that comprised a total of 112 participants and included 19 individuals who acquired EHEC infection were questioned regarding the foods they consumed after eating in a common restaurant. Additionally, the menus ordered by the participants were identified by means of order lists and meal receipts. The restaurant kitchen was questioned in detail regarding the preparation and the type and quantity of ingredients in each menu ordered by any of the study participants. Furthermore, available photographs taken by travel group members were analyzed to confirm which food items, including toppings, were seen on the plates. The data thus gathered was analyzed in a cohort approach that permits the retrospective estimation of the relative risk of infection for the restaurant customers. Results of this analysis showed that customers who ate sprouts had an 8.6-fold increased risk [of] illness compared to those who did not. This study also revealed that 100% of those who contracted the illness had eaten sprouts.”

The decision process was made all the more difficult because it appears some records were held on paper rather being completely computerized, and health authorities were sometimes communicating with hospitals and each other via letters in the post rather than electronically, and were having trouble tracking who had become ill.

An overly-manual and confused decision-making process hindered the discovery of the “sproutbreak” that lead to so much suffering and hardship. But at least some good has come out of it: a better process, and a more detailed questionnaires will now be a standard part of outbreak investigation in the future.

All organizations need to be able to document and improve their decision-making processes, even if (thankfully) it’s rarely a case of life or death. Technologies like SAP Streamwork are starting to allow companies to apply standard templates to frequently-occurring decisions, and collaborate to come to the best conclusions, using an on-demand platform.

Could you improve decision-making in your organization by optimizing decision collaboration?

Other Links

• http://scienceblogs.com/aetiology/2011/06/german_officials_declare_e_col.php
• http://scienceblogs.com/thepumphandle/2011/06/culprits_in_germanys_foodborne.php
• http://www.rki.de/EN/Home/homepage__node.html

Here’s an example from the video, where somebody claims in a blog post that global temperatures have been dropping, using this straight-line “analysis”:

As the video makes clear, to work out what’s really happening to the average global temperature, you have to do sophisticated statistical analysis to adjust for the cyclical trends and/or doing a rolling average.

But looking at the longer-term data set, it should have been pretty obvious that 1998 isn’t a great year to start the comparison from (unless you’re a willful global warming denier?)

The ramblings of these self-declared experts are then picked up and promoted throughout “the internets”, and end up being widely believed (a good example of confirmation bias). Thankfully, journalists like Peter and web sites like Ben Goldacre’s excellent Bad Science blog do a great job of trying to minimize the damage caused by this disinformation.

Sadly, though the same thing happens in organizations, too – bad data is too often badly analyzed and simplified, then passed on as “truth” to the rest of the organization. Just like on the internet, some people are happy to jump on data that confirm their existing biases, and may not be keen to figure out whether it’s actually true or not (a former boss, Dave Kellogg, once told me the first question to ask any executive who requests a research study is “do you want a high number or a low number?”).

This problem will never go away, since it’s firmly anchored in the weaknesses of human psychology. The best long-term remedy is to have pervasive business intelligence (so there’s less reliance on anecdotal evidence), to employ information analysts (the equivalent of Peter and Ben) in a BI competency Center, and to expose bad analysis to more people using collaborative decision-making environments like SAP StreamWork.

There are only three matches left in this year’s World Cup, and the competition is wide open after a series of upsets have knocked out some of the big international players, including the winners of the last three world cup tournaments: Italy, France, and Brazil.

To enhance your enjoyment of the game – or simply to see a real-life example of what you can do with business intelligence technology, check out the suite of tools offered as part of the SAP BusinessObjects 2010 Football Experience:

The site lets you:

• Easily review historical data from matches going back nearly 50 years to gain perspective on the game.
• Check out details on the teams and players as they compete for the title.
• Analyze match information, such as shot location, to search for patterns of play and success.
• Gain insight, spot trends, and use the information you uncover to make better decisions.
• Receive live commentary during the matches so you can stay informed.

In order to get answers to questions such as:

• What positions typically score the most points?
• What is a likely final score?
• Does being the host country influence success?
• Which teams consistently make it to the final rounds?
• Which players have previous tournament experience?
• Is there a particular area of the goal that a goalkeeper is less successful in defending?
• Is one player or team receiving more yellow cards than the others?

Trivia contest

Once you have the hang of the different tools below, why not take part in the Trivia Contest? Each week day a game is on, there will be a question tweeted about from the @businessobjects Twitter account, and the first correct response is the winner!.

The first thing to do is register with the site (you can still access some of the functionality without doing this, but you won’t get the full flavor).

You can then go to the main screen and choose which of the seven main areas you’d like to explore:

• Go Explore! World cup statistics with SAP BusinessObjects Explorer
• Predictor: Predictions using SAP BusinessObjects Xcelsius
• Goal History: Reporting using SAP Crystal Reports
• Social Network Analyzer: The relationship between teams, matches, players…
• Team Information: An Xcelsius dashboard of team information
• Goal History by team: Reporting using SAP Crystal Reports
• Jive community space: Join in the conversation!

Go Explore!

Click the Explorer button to open up a choice of information spaces: either the full statistics, or a view of other people’s predictions (see next section) (note, if you only see one choice, check again that you’re registered / cookies are enabled)

Click on the first one and start investigating – e.g. one of the semi-finals is Germany vs. Spain – here’s the result of the previous time the two teams met:

There’s also a set of shortcuts that you can use to get an idea of what kinds of questions are possible:

• What unlucky player has hit the goal post on 4 separate occasions in a single World Cup tournament? ANSWER
• Jürgen Klinsmann is an former player who is now working as an ESPN commentator. When did he play, and how many goals did he score? ANSWER
• What Italian goal keeper has had a remarkable 69 saves in World Cup play? ANSWER
• I heard that Real Madrid is a good club. How are the players from that team in this year’s World Cup?
  Has anyone scored? ANSWER

Predictor

Using Xcelsius, you can predict the scores of the different matches, and then see how good your prediction record is over time:

And then you can access a leaderboard report that gives you the top predictors, overall, or by round, as a pdf document:

(and remember, you can do exploration of the same data by opening the other information space in the explorer section above).

Goal History

The goal history section opens up a Crystal Report with detailed data about exactly where and how goals were scored during matches:

And you can drill down on information by different teams – here’s Germany’s scoring pattern in the last world cup, for example – something that Spain might like to look into before their next match…

World Cup Social Network Analyzer

The SAP Social Network Analyzer aggregates existing enterprise data to display and discover organizational relationships. It automatically generates useful social networks that can be used:

• to find and connect people,
• to take actions based on individual/organization/company information,
• to send an email, meeting request or call a person,
• to build the right team,
• to better manage and control processes,
• to understand the relationships between suppliers and buyers,
• to analyze people’s information and organization using BI tools,
• to integrate social network information inside any application…

All the data from the world cup has been fed into the online prototype, to give you an idea of how the players and their teams relate. For example, here’s the network of matches played by the USA over the years:

Here are some same views:

• Player with current national team, coach, number of goals in world cup, size, height, position or jersey number
• Teams with their respective players and coaches
• Players who scored during a world cup still in 2010.
• Who scored the most
• Previous world cup matches between two teams

Team Information

This section gives you access to an interactive Xcelsius dashboard that shows information about the different teams, e.g. by location:

And by previous team experiences in the world cup:

And the players involved:

Goals per Team

Another example of a Crystal Report, this time using the Crystal Reports Viewer plugin.

Jive Community Space

Join in the discussion! Will the Dutch be able to set the pace in the next match?

Other Dashboards

Other folks have also taken the plunge and used the SAP BusinessObjects tools to create useful dashboards… here’s a nice one from the MyXcelsius.com web site, using the great Google Maps 2.0 plugin for Xcelsius:

SAP Football Ad:

If you’re into football, you’ll enjoy this SAP Ad from 2008, courtesy of www.danielfreestyle.com (filmed entirely around San Francisco, which does a pretty good job of looking like a trip around the globe!)

SAP BusinessObjects Explorer is the first analysis tool that I’ve seen that I can honestly say is intuitive enough to use without any training (see http://microfinance.sap.com to prove it to yourself).

If you’d like to know more about the product, list of “Everything You Ever Wanted to Know about SAP BusinessObjects Explorer” in a previous post.

This post is about the http://explorer.ondemand.com web site that lets you easily upload any Excel spreadsheet and start analyzing, for free, without having to install any software, and the updated version of the Explorer in the Cloud prototype that has just been released by the SAP BusinessObjects innovation center.

New Features

The new version uses the latest interface of SAP BusinessObjects Explorer XI 3.1 SP1, and adds two key functionalities asked for by early users:

Analysis bookmarking. Once you’ve discovered something in the data, you can now easily send that view of the data to somebody else. You simply click on the “Email” button, you’ll be prompted whether the view should be password-protected, and whether you’d like it to be a permanent bookmark. For example, you may want to make a breakdown of product sales permanently available to the marketing team.

Replace dataset. The new version lets you replace an existing data set with an updated one without disturbing any existing links or bookmarks.

And, of course, you can still easily export the analysis as a data set or image:

“Drinking our Own Champagne”

The solution is not yet available externally on the http://explorer.ondemand.com website, but since we’re big believers in “drinking our own champagne”, it has already been made available internally to all 48,500+ SAP employees as part of the standard internal portal.

Feedback has been extremely positive across all communities and geographies, with hundreds of sessions every day. Here’s one feedback, about a typically pragmatic information need (paraphrased and translated from the original):

“I’m in a technical team. Each region created a spreadsheet containing information about the skills and levels of their local team members. Using the internal Explorer solution, it was easy to consolidate the different spreadsheets, upload them, analyze the data to spot areas we needed to work on, and get the information into PowerPoint so we could share it with others!”

Excel Plug-in and API

An Excel 2007 and 2003 Add-in is available that lets you send information directly from your spreadsheet environment – select the data you want to look at and you can start analyzing just two clicks later.

Finally, an API for Explorer in the Cloud is also available – think about ways you or your organization could ship information directly into the cloud for easy analysis!

If you’d like to know more about Explorer, here’s a new promotion page that gives you links to the latest resources about Explorer.