A BI Self-Service Debate: Can Users Be Trusted?

At SAP TechEd in Amsterdam this week, I had a conversation with SAP mentor Steve Rumsby about BI self-service and user trust.

You can get an idea of our discussion below, which echoes some other conversations I’ve had with IT organizations that are worried about potential bad analysis that could result from unfettered access to data. I’m very interested in your feedback: chime in with your thoughts and comments!

Steve, before we start, what’s your background and role?

I’m the SAP technical manager at the University of Warwick in the UK. I’m formally responsible for our technical architecture, servers, storage, network, plus the development work. We run SAP ERP 6, GRC, solution manager, plus other peripheral systems. I joined the IT department directly after finishing my computer science degree at the university.

So what do you think about BI self service?

I wrote about this on the SAP community network. I can see why people get excited about it, but it makes me feel nervous. I think it’s a double-edged sword. There are lots of good things that you get from it – people able to get information themselves, without having to wait weeks, and so on. But what makes me nervous is people doing that without understanding the nature of the data they are trying to manipulate.

Any examples?

There have been some famous ones – for example, one respected UK newspaper reported that there were only a handful of adult cod left in the North Sea.

The BBC did an investigation, and the result was that they were wrong by a factor of 1000000. The misunderstanding was based on comparing life-time information from cod in a different part of the world, and making unwarranted assumptions… if you don’t understand fish, you won’t realize you’ve made a mistake until you publish. If that was the organization’s cash flow, people might not be so happy!

But surely the problem was this was a newspaper, with no expertise — if it was the cash flow, wouldn’t the analysis be done by finance people, who do know what they’re talking about?

Yes, but what gets people really excited is the ability to take data from all different places, and mash it up. So you start with data you do understand, but you mash it with data you don’t necessarily understand, and you get an answer that you believe, but maybe you shouldn’t. How do you know if it’s false?

So is the answer to force business people to always go through IT experts? That has resulted in huge frustration, and business people creating shadow IT organizations…

The answer is that people need to understand what data they can trust and what they can’t. It’s about education and training: as soon as you go and use outside data, you must think harder about the trust-worthiness of the data and the analysis.

But when I look at organizations that are keen on “self-service”, it’s typically not about external data. It’s more about fast-changing or manually sourced local or departmental data doesn’t make financial sense to put in the data warehouse (e.g. for a one-off analysis) — yet some organization’s BI strategy is still focused on the DW, so the users are stuck… what happens then?

In those cases, yes, users should be able to go ahead and do it, but they need to speak to somebody who can help them understand and sense-check the data.

I believe this is one of the critical roles of a BI competency center: it’s not just about owning an infrastructure, but owning the general business problem of helping business people with their analysis, no matter what the tools used…

I agree…

Further reading:

• Steve’s original blog post on “Is BI Self-Service Really A Good Thing?”
• A reply from Gareth Ryan, “Self-Service BI, or Seeing Cod On The Moon”