Business People Are Dumb On Average(s)

Here’s one real-life example of the limits in quantitative thinking observed in an average company, shared with me by Michael Thompson. He recently worked with Dan Rosenberg’s UX team on a SAP research project that isolated over 100 distinct, real-life uses of business analytics across dozens of roles and business functions.

A retail analyst was attempting to figure out the success of product promotions on the “sales” page of a web site. After gathering the data, he used Excel to create the table below. He compared the increase in each product sold with and without the promotions, and then tried to calculate the average percentage increase across the different products. The result was that “the promotions increased sales by 151% on average”.

This result was then used to create a rule-of-thumb of how much extra inventory the company should have on hand at the start of a promotion: “roughly double”. This lowball figure, it was felt, would generally provide enough stock for promotions, while reducing the chances that the company would be left with too much stock at the end of the promotion.

But there are a whole bunch of problems with this analysis. The most basic mathematical problem is that, while there are exceptions, calculating the average of percentages almost always gives an unwanted answer.

A better way to calculate the success of the promotions is to compare the average sales of all the products before and after a promotion (see table below). This shows that the promotions increased the average quantity of products sold by only 71% – considerably less than double.

* note that it would be it would be easier to compare the % increase in totals items sold, which would come out to the same result, but I’ve used the averages to make the comparison between the two methods as obvious as possible.

So if the company did use the proposed rule of thumb they might have a nasty shock. Instead of an expected “lowball” inventory that would be used up by the promotions, they should instead expect to have an extra 29% of the original items left over at the end.

And beyond the numbers, there’s a lot of unanswered questions about the analysis that call into question its usefulness:

• Is units sold the best measure? Wouldn’t it be better to work with revenue or profit?
• There’s huge variability between the different categories of product. Is the notion of an average even meaningful? Maybe categories of products could be created for more meaningful analysis?
• Do these percentages stay reliable over successive promotions, or are they also very variable?
• Is it in fact better to have too little stock than too much? By how much?
• Might external variables or seasonal variations have lead to the observed results, rather than the promotion? (e.g. maybe it was the 4th of July, and charcoal sales would have soared anyway)

In general, people are tempted to throw up their hands at this point and say “well, we have to make some sort of prediction for inventory levels – isn’t this better than nothing?” And the answer is yes (as long as you get the basic math right!), but somebody in the company should be aware of how limited the analysis currently is, and be constantly trying to improve it. Unfortunately, as in this case, real-world business people often rely on basic, faulty analysis, and lack the curiosity (or incentive?) to push it any further.

Given human nature, it’s probable that this example is not an isolated case. If you really want to improve your company’s return on investment in analytics, you may want to consider investing in more training rather than yet another technology solution.