Business People Are Dumb On Average(s)

perplexed-banner

Here’s one real-life example of the limits in quantitative thinking observed in an average company, shared with me by Michael Thompson. He recently worked with Dan Rosenberg’s UX team on a SAP research project that isolated over 100 distinct, real-life uses of business analytics across dozens of roles and business functions.

A retail analyst was attempting to figure out the success of product promotions on the “sales” page of a web site. After gathering the data, he used Excel to create the table below. He compared the increase in each product sold with and without the promotions, and then tried to calculate the average percentage increase across the different products. The result was that “the promotions increased sales by 151% on average”.

This result was then used to create a rule-of-thumb of how much extra inventory the company should have on hand at the start of a promotion: “roughly double”. This lowball figure, it was felt, would generally provide enough stock for promotions, while reducing the chances that the company would be left with too much stock at the end of the promotion.

Product Average items sold prior 3 weeks Items sold during special promotion % increase
Kams Mint Toothpaste 8 oz 72 112 56%
Peepers Size 5 Diapers 32 pack 134 170 27%
Pata Negra Ham Sandwich 35 43 23%
Closers Breath Mints 40 112 180%
Bboy Barbecue Charcoal 2lbs 17 98 476%
Lindas Cookie Ice cream kids treats 26 65 150%
Giant Corn Chowder Soup 12 oz can 43 84 95%
Silly String Cheese, Lunch pack 12 55 358%
Green Label 6-pack beer 120 115 -4%
Average of % increase in quantity 151%

 

But there are a whole bunch of problems with this analysis. The most basic mathematical problem is that, while there are exceptions, calculating the average of percentages almost always gives an unwanted answer.

A better way to calculate the success of the promotions is to compare the average sales of all the products before and after a promotion (see table below). This shows that the promotions increased the average quantity of products sold by only 71% – considerably less than double.

Product Average items sold prior 3 weeks Items sold during special promotion % increase
Kams Mint Toothpaste 8 oz 72 112 56%
Peepers Size 5 Diapers 32 pack 134 170 27%
Pata Negra Ham Sandwich 35 43 23%
Closers Breath Mints 40 112 180%
Bboy Barbecue Charcoal 2lbs 17 98 476%
Lindas Cookie Ice cream kids treats 26 65 150%
Giant Corn Chowder Soup 12 oz can 43 84 95%
Silly String Cheese, Lunch pack 12 55 358%
Green Label 6-pack beer 120 115 -4%
Average quantity 55.4 94.8
% Increase in average quantity* 71%

* note that it would be it would be easier to compare the % increase in totals items sold, which would come out to the same result, but I’ve used the averages to make the comparison between the two methods as obvious as possible.

So if the company did use the proposed rule of thumb they might have a nasty shock. Instead of an expected “lowball” inventory that would be used up by the promotions, they should instead expect to have an extra 29% of the original items left over at the end.

And beyond the numbers, there’s a lot of unanswered questions about the analysis that call into question its usefulness:

  • Is units sold the best measure? Wouldn’t it be better to work with revenue or profit?
  • There’s huge variability between the different categories of product. Is the notion of an average even meaningful? Maybe categories of products could be created for more meaningful analysis?
  • Do these percentages stay reliable over successive promotions, or are they also very variable?
  • Is it in fact better to have too little stock than too much? By how much?
  • Might external variables or seasonal variations have lead to the observed results, rather than the promotion? (e.g. maybe it was the 4th of July, and charcoal sales would have soared anyway)

In general, people are tempted to throw up their hands at this point and say “well, we have to make some sort of prediction for inventory levels – isn’t this better than nothing?” And the answer is yes (as long as you get the basic math right!), but somebody in the company should be aware of how limited the analysis currently is, and be constantly trying to improve it. Unfortunately, as in this case, real-world business people often rely on basic, faulty analysis, and lack the curiosity (or incentive?) to push it any further.

Given human nature, it’s probable that this example is not an isolated case. If you really want to improve your company’s return on investment in analytics, you may want to consider investing in more training rather than yet another technology solution.

Comments

One response to “Business People Are Dumb On Average(s)”

  1. […] focusing only on advanced analytics, we need to encourage all employees to be more data literate (see this example of what can go wrong) and encourage more shared […]