Fixing the BI Tragedy of the Commons

Cindi Howson talks about the “BI Tragedy of the Commons” in her latest Intelligent Enterprise post.

The “tragedy of the commons” is a well-known issue in economics, involving a conflict between what’s good for the individual and what’s good for a group as a whole. Cindi uses it to describe the problem of departmental BI silos when it would make more sense to use a more enterprise-wide approach.

One problem with this example is that once you take all the variables into account, it’s not clear that silos are necessarily a bad choice, at least in the short term. My experience is that trying to take an enterprise approach too early is rarely a good solution in the real world, where it’s typically better to develop a series of departmental projects first, and only then go back and develop a more information infrastructure approach.

But I agree 100% that the “tragedy of the commons” does indeed plague business intelligence deployments. Luckily, the potential remedies are as well known as the problem — although they can be difficult to apply.

Put simply, economics is the study of incentives — and my experience is that misaligned incentives are one of the biggest issues facing business intelligence today (see related post on who “owns” BI issues)

To fix the problem, you have to assign ownership rights to the assets in question. For example, you can “enclose the commons”, and force people pay if they want to let their animals graze. The land doesn’t get over-grazed because people aren’t willing to pay to get access to land that no longer provides any nourishment.

By default, most organizations give ownership of their BI projects to the department that implements them — i.e. they have the “right” to create silos. This is the equivalent of farmers having the right to let their animals graze, and the opposite of the example above.

So one possible solution to our problem is to reassign ownership rights to a special body designed to get the best use of the organization’s assets — such as a BI competency center. If a department wants to build a silo application, they would have to “pay” the competency center for the right to build a silo — or at least explain why it was the best course for the business as a whole, rather than just their department. Note that this could still result in some silos being built, but only when the circumstances made sense.

However, having a central, “ivory tower” body controlling departmental activity is the inverse of today’s trends in decentralization. More and more organizations are pushing decision-making out to the organizations on the front-line, recognizing that the increased ability to react fast to new situations typically trumps the increased efficiencies of more centralized structures.

Luckily, in classical economic theory, you get optimal use of resources whoever gets ownership rights. So we can address the problem differently: departments can retain the “right” to make silos, and the BI competency center can “pay” the departments to do the right thing for the organization as a whole.

This solution is much better aligned with the way organizations typically work. The BI competency center get funding through a “tax” (e.g. the “IT infrastructure” line in the annual budgeting cycle) and provides expertise and resources to departments that agree to take a more enterprise approach.

Note that this situation allows for an “optimal” mix of silos and enterprise-wide solutions — if a department considers that they can build a valuable silo without the central resources, they’re still free to do so. By controlling the level of “tax”, organizations can exert more or less centralized control.

The problem is typically that organizations don’t think in terms of ownership rules, and fudge an incomplete and confusing mix of the two approaches, with some centralized control battling against the departments desire for autonomy, and disagreement about “what’s best for the company”. This in turn results in tension between IT and the business, further decreasing the likelihood of long-term BI success.

To summarize; First, it’s essential that organizations have a body that has the authority to represent the “common good” of integrated information — such as a BI competency center. Second, it has to be clear where “ownership” lies, and what mechanisms exist to “pay” the owner to do the right thing for the organization as a whole.