Data Democracy vs. Data Anarchy: Governed Data Discovery

Successful business intelligence has always been about negotiating the right tradeoffs between the needs of individual business people and the needs of the organization as a whole.

In 2001, Bernard Liautaud, founder of self-service BI pioneer BusinessObjects  wrote a book called eBusiness Intelligence: Turning Information Into Knowledge Into Profit that discussed these tradeoffs. He drew the analogy with systems of governments, illustrated by an information governance quadrant that outlined the choices between Information Dictatorship, Information Anarchy, and Information Democracy*.

BI technology has taken a big leap forward since the book came out a decade ago. But the tensions between IT and the business and need for tradeoffs have remained, and we’re now seeing the latest generation of analytics struggling with the same issues.

Much of the recent growth in the business intelligence market has been in the area of “data discovery” tools that take advantage of in-memory technology to allow users to quickly mash up data from multiple sources and explore it interactively and visually without having to create new “queries” for each step.

These tools are often enthusiastically adopted by business users as a reaction against the red tape and restrictions of central IT organizations, which are perceived as information Dictatorships where access to data is concentrated in the hands of a few.

Although this data discovery has empowered business users in new ways, it has also introduced new dangers, in the form of Information Anarchy as defined by Liautaud:

Information anarchy results from individuals or entire departments taking their informational needs into their own hands. As businesses grew more competitive in past years, departmental managers realized they needed better information to make good business decisions. They realized that whatever information they could procure from the glass house of the IT department would not be adequate.

One common side-effect of such tools is new data silos, where each group or department compiles its own figures, resulting in inconsistent data and duplicated efforts:

Similar to the brief yet short-lived instances of anarchy in world history, the onset of information anarchy is often followed by a brief period of euphoria. Departments that set up their data silos are briefly enthusiastic about their limited solution, until the point in time when someone in senior management receives two reports with inconsistent data from different departments and questions the validity of the information. The uncoordinated silo approach sooner or later will result in data consistency issues, while little security enforcement on data access is feasible.

Even worse, “data fiefdoms” can arise that replace central IT with more local bottlenecks. Some call this creation of a new “information aristocracy” — and it’s hard to read all the praise for new “Data Scientist” roles without worrying that we are creating a new layer between business people and the data they need for decisions. As Liautaud put it a decade ago:

Executives and other business managers “beholden to number-crunching wizards specially trained with reporting, analytic, and statistical software installed on their computers. The concept was to expand the use of information to many more business users… however, very quickly, this technology-schooled cadre of individuals became, without knowing it, information dictators of another sort.

Businesses should strive to combine consistent information governance with decentralized access to data and decision-making. Liautaud called this “information democracy,” and it’s similar to what Gartner now calls governed data discovery: platforms that address both business users’ requirements for ease of use and enterprises’ IT-driven requirements.

According to Gartner, the race is now on between newer data discovery vendors and traditional vendors to provide these platforms.  The customers of data discovery vendors:

“would like to expand use and even standardize on data discovery platforms for their larger enterprise BI deployments but find that in many cases these platforms …lack the necessary enterprise features in relation to governance, administration and scalability, among other things.”

And in the meantime:

“the current IT-centric BI and analytics platform Leaders, which currently “own” the installed base market share… are trying to change that by focusing their new product investment on business-user-driven data discovery and analysis.”

For example, SAP, the clear leader in the business intelligence space for many years has introduced the data discovery product SAP Lumira. A desktop version of the tool allows analysts to mash up data and create visualizations, then easily share them via the existing SAP analytics platform. This allows organization to support a “network of truth” or “collaborative insight” without sacrificing either the ease of use needs of the individual or the data governance needs of the organization as a whole.

Of course, any form of democracy is hard, precisely because it requires constantly updated tradeoffs between different groups. The key is to build a system that everybody respects, even if they don’t always like the result. This in turn requires that both IT and business people have to realize that some compromise is necessary. For example:

• Giving business users more freedom to access and use information may result in more mistakes and duplication than IT is comfortable with, but the alternative is often worse: business people making decisions with inadequate data because they can’t wait for slow IT processes.
• Business people must realize that optimizing only their own information use and access can damage the ability of the organization as a whole to get a clear picture (this is the “BI tragedy of the commons“)

Finally, it’s always about the business needs, not the technology platform. Liautaud’s book gives some timeless advice to anybody struggling with their business intelligence vision:

Buzzwords may come and go and technologies evolve. However the key process remains the same: employees need to access, analyze, and share data on their business, in order to support their decisions using factual information. The following five strategic recommendations should be kept in mind by business managers as their company builds a business intelligence system:

1. Start with the business
2. Define the data architecture in a joint effort with the business users and IT
3. Develop a business intelligence strategy with the user in mind
4. Build it to grow
5. Think beyond your borders

_______________________________________________________________________________

* Howard Dresner, the former Gartner analyst who coined of the term “business intelligence” in its modern sense, used a similar model in his 2006 book The Performance Management Revolution