The term OLAP or Online Analytic Processing was coined in 1993 by relational database technology pioneer Ted Codd (my claim to fame: we went to the same high school, Poole Grammar).

The term was chosen to contrast with OLTP or online transaction processing, and was prompted by some clever marketing folks at Essbase, who wanted to promote their multidimensional database product. Codd was famous for his twelve rules defining the relational model and duly came up with twelve rules for analytic systems.

The term was quickly taken up by the rest of the industry, and spawned new definitions (Nigel Pendse’s FASMI test) and multiple variations (MOLAP, HOLAP, ROLAP, Huey, Dewie and Louie, etc.).

Over time, these multiple definitions started muddying the meaning of the term (was it a technology? a user interface? an approach to analysis?), and Gartner decreed that it was ‘just’ part of a larger market called business intelligence. The result has been a long slow decline of the use of the term OLAP, as the Google Trends chart below indicates.

Only Nigel Pendse of the OLAP Report tried to side-step this trend, and continued producing OLAP-specific analysis for many years, but “business intelligence” was clearly the mainstream industry term. A few years ago, Nigel sold the OLAP Report to the German BARC group, who initially continued under the same name, and tried vainly to convince everybody that OLAP was still a “hip term”, but finally succumbed to the inevitable and announced last month that they would be changing the name of the report/site to The BI Verdict.

(Sadly, at some point in this process, BARC decided to lock one of Nigel’s best articles — “How not to buy a BI product” – behind their subscription firewall. All I can find on the web is a far-less-entertaining summary of the main points here. [UPDATE: thanks to a tip from Florian Bahr, I can point you to the full article on the WayBackMachine: How not to buy an OLAP product)

Since BARC were the last group using the term with any frequency, it’s now fairly safe to say that OLAP’s days are over, but interestingly, the group seems to have chosen to shift to the wrong term. The chart below shows that the search trend for “business intelligence” has been slowly drifting down over the last five years.

And the decline is even more pronounced for another standard industry term, “performance management”

Since BI and performance management remain fast-growing markets, this trend is a little surprising – until you look at the search figures for the term analytics. Starting in 2005 (perhaps prompted by the introduction of Google Analytics?), the term has skyrocketed in use.

Possible reasons for this may include:

• Popular books and articles aimed as business people tend to use the term, such as Thomas Davenport’s 2007 book “Competing on Analytics”. This is perhaps because there’s more ambiguity for a business audience, who associate the term “business intelligence” with industry data vendors like Reuters and Thomson (now both part of the same company).
• The acquisition of the mainstream BI vendors by larger organizations (Hyperion by Oracle, Cognos by IBM, and BusinessObjects by SAP) has meant that the industry has been increasingly using other, more generic terms, such as “embedded analytics” and “analytic applications” to explain the same functionality. And the largest remaining independent vendor, SAS, has proclaimed themselves the leader in “business analytics”

My conclusion? By the time you read this, this blog might well be called “Analytic Questions” instead of “BI Questions”…

IDC recently released several reports on business analytics market shares, including business intelligence and performance management:

• Worldwide Business Analytics Software 2009 – 2013 Forecast and 2008 Vendor Shares, IDC #219383, August 2009
• Worldwide Financial Performance and Strategy Management 2008 Vendor Shares: Market Consolidation Drives Domination, IDC #218656, June 2009
• Worldwide Business Intelligence Tools 2008 Vendor Shares, IDC #218598, June 2009

You’ll have to subscribe to IDC’s excellent service to get the full details, but if you look carefully through industry articles and vendor samples of the reports you can get many of the key figures and information about how IDC divides up the analytics market.

Here’s my interpretation of the big picture trends:

1. It’s a race between Oracle and SAP. IDC divides the analytics market into three main areas, all roughly the same size (there is also a fourth area, Spatial Information Analytics, but this makes up only around 3% of the overall market)

• Performance management and analytic applications (SAP #1)
• Business intelligence tools (SAP #1)
• Data warehousing platform software (Oracle #1).

Both Oracle and SAP are growing faster than the market average, implying consolidation around the market leaders (although some of the top growth figures came from some of the smaller vendors in the space).

2. Oracle’s lead is narrowing, with SAP is coming on strong… Oracle leads the overall analytics market by IDC’s definition, but Oracle’s growth has slowed since last year, while SAP’s has increased. Oracle is losing market share in its traditional area of strength (data warehousing platforms) where the #2 and #3 (IBM and Microsoft) both grew faster. SAP is growing faster than Oracle overall and in each of the three main areas.

3. …Particularly in FPSM. Thanks to the acquisition of Hyperion, Oracle has been the leader of what IDC calls the Financial Performance and Strategy Management Applications market (part of the analytic applications area) since 2006. But some key acquisitions by #2 SAP (including Pilot and Business Objects) have narrowed the gap considerably, and SAP experienced almost 3x Oracle’s growth over the last year.

Are the Numbers Accurate?

It’s human nature to assume that any set of numbers is more trustworthy than simple rankings. But just how accurate are the numbers, really?

IDC puts a lot of effort into gathering and interpreting the market share figures, and takes a lot of care to allocate the total vendor revenues among the various markets in which they participate. Despite this, there are at least three reasons that the numbers they provide are necessarily inexact.

• First, the numbers can never be more than IDC’s “educated guesses.” Companies that cover multiple market segments almost never publicly break out revenue according to IDC’s groupings, and so are not allowed to provide them selectively to IDC. IDC works with the vendors to try to ensure that there are no obvious mistakes, but there’s an incentive for vendors to try and influence the numbers in their favor (e.g. they may try to persuade IDC that some of the revenue from an area where they are a clear leader should instead be allocated to an area where they are the #2, making them the #1 in both markets).
• Second, even the vendors may only have a hazy idea of the exact revenue that comes from each area. Companies typically purchase related products from vendors in a single deal, and then haggle over the discount amounts. And a single vendor “product” may bundle elements from several different IDC categories. The allocation of the final deal amount between the different components (for example, a database, a data warehouse, some BI tools and some financial planning and budgeting tools) can vary over time, and may be different for different companies. Public companies have to respect some limits set by the SEC concerning revenue recognition, but private companies have a lot of discretion when it comes to stating their revenues (and this does not just concern the smaller vendors — SAS, the #4 vendor overall, is privately held).
• Third, different vendors sell through different channels. If a company sells directly to the final customer, the market share is the same as the revenue, but if it sells through resellers (who make a margin), the real market value of a product may be under-represented by the revenue reported by the vendor. This isn’t technically an inaccuracy, since IDC is explicitly measuring vendor revenues, but it makes it harder to interpret what the numbers actually mean.

Overall, these are marginal concerns, and the overall rankings are probably correct, but I would love to see IDC provide estimates of margins of error directly within the reports, making it easier to know whether small differences are in fact important.

Does Top Market Share Mean that the Vendor has the Best Products?

What do the market share numbers actually mean? What do they imply? What action could or should be taken based on them?

Obviously the vendors and their shareholders care about who’s selling the most (although even then share price typically depends more on profitability than overall revenue). But should anybody else?

Typically, a claim of having the top market share is used to imply that the products are the best ones available in the market (and that is why they are selling so well). But is it true?

Well, yes, they probably are – but in the same way that the #1 in the music charts represents the “best music”.

Just as the top-selling music has the broadest appeal, the leaders have the most broadly useful products. However, just as your own musical tastes might involve something more niche, it may be that a niche BI product or analytic application may be the best for any particular project (which is why there will always be small, focused vendors in the space).

However, the analytics choice most organizations have to make is akin to having to choose the music that will most appeal to all of the different employees in your organization, rather than the music you personally like best. This means most organizations probably shouldn’t choose niche products, because of the value of being able to combine information across different project silos, and because people need multiple different types of access to information.

For example, a sales manager may need a dashboard to track key sales metrics, an OLAP tool for budgeting, an analysis tool for investigating the sales pipeline, and some regular HR reports about team vacation planning. And she would like the data in these reports to be consistent from one usage to another, even when the data in the reports comes from multiple different systems and platforms.

Different products from different vendors may be the “best” niche solutions for each of these needs, but forcing the sales manager to use four unrelated systems, and cope with the inevitable data differences between them, is unlikely to be the best real-life solution. And of course, it’s typically more expensive to deploy multiple solutions than different front ends from the same vendor.

It’s also worth noting that the major vendors may also provide the best niche solutions in many cases, since they typically became major vendors by having the best products in the first place, or grew through the acquisition of other best-of-breed vendors.

So one reason that vendors emphasize market share is because it’s at least somewhat correlated with product quality. Most organizations do a thorough evaluation of the tools available in the market before choosing, and the majority of transactions are extensions to existing deployments, so market share is in some ways the most extensive “customer satisfaction survey” you will ever see. And rather than just ticking boxes, companies and individuals actually have to back up these judgments with money and their reputation.

Other Advantages of Market Share

In addition, choosing a product from a larger vendor provides some tangible benefits above and beyond the quality of the products themselves:

• More support for a wider set of architectures and interfaces. There are economies of scale for supporting different platforms, and larger vendors are likely to have more resources for keeping up with the latest versions of the systems that make up your core information environment, and tighter technical relationships (such as getting earlier access to the next generation of new products). Smaller vendors are typically forced to concentrate on a particular platform or environment. This is of course fine if your environment is supported, but not if you have a complex environment, or may need to change in the future.
• Worldwide services and support. Larger vendors are more likely to be able to support your local language and support requirements. Again, this would not apply to a small vendor in your country who is working in your language, but may be a problem if you need to span geographic locations now or in the future.
• Vendor ecosystem. The success of BI projects are about more than just technology. Larger vendors typically have a more extensive network of consulting and training partners, more other companies you can talk to about using the tools, and higher availability of employees with skills in the vendor’s technology.
• Less risk. The market is undergoing consolidation, and smaller vendors are more likely to go out of business, or be acquired.

There are certainly factors that could also argue in favor of smaller vendors. For example, they may be more responsive, since you represent a larger part of their business, or more innovative, since they have a smaller installed base to maintain, and a stronger incentive to innovate in order to be able to win deals against the bigger players. And they may be more open, with less incentive to tie you to additional products in the vendor’s portfolio (but by the same token, they make be less integrated with the other tools that do you have).

On average, software markets typically consolidate around a small number of larger organizations, showing that in the long run the benefits of size outweigh the potential downsides. And the fact that purchasers don’t want to take “risks” with smaller vendors and that this “winner takes all” trend is firmly embedded in the consciousness of corporate buyers everywhere makes it even more of a self-fulfilling prophecy.

Which Vendor is the Leader?

What does “market leadership” refer to? Is the “leading” product the one with the biggest market share? Or the highest quality? Or the the highest growth? Or some combination of the above?

Clearly, the term is open to interpretation, and vendors often spend considerable effort to find a marketable leadership claim – even to the extent of trying to tailor or invent market segments in which they are the leader (for example, Arbor software essentially invented the “OLAP market” as part of their marketing plans).

Despite much cynical eye-rolling from customers, it seems clear that these claims will always be part of the marketing landscape, since most of us do indeed want to be able to say we chose the “best” or “leading” software for our organizations.

For example, Oracle places ads such as the one on the right in magazines and airports around the world, claiming leadership in Enterprise Performance Management.

But SAP insists it is the #1 in the space, based on its leading share of the analytic applications market, which includes financial planning and strategy management (IDC does not use the term EPM).

According to IDC’s figure, Oracle is not #1 in business intelligence, and probably not in operational analytics (since SAP is leader in both applications and analytic applications). So what is the basis for the claim in the ad?

Oracle provides a link to a web page that attempts to back up the assertion, and which doesn’t mention market share at all. Instead, it links to reports from analyst companies like Forrester and Gartner that use more qualitative measures of “leadership” (interestingly, SAP could probably use the same list of reports to argue it was the overall leader).

Conclusion

Overall, market share is an important criteria that should not be ignored in purchase decisions. You can absolutely have great analytics projects with tools from smaller vendors. And successful analytics is much more about people and process than it is about technology, so in theory you can succeed with any tool.

But the whole point of analytics is that it must evolve over time, and adapt quickly to any changes in your business, and integrate with other systems inside and outside your organization, and I believe this gives the advantage to the larger vendors.

Among the larger vendors, does it matter which is #1 and #2? To SAP and Oracle, yes. For everybody else, it’s just one factor to consider, along with functionality, fit with your architecture, and – of course – cost.

Oracle’s recent acquisition of best-of-breed performance management vendor Cartesis means that ERP vendors want to take their BI-related functionality up a notch.

Wait! Didn’t Business Objects buy Cartesis?! 

The article should of course read “…performance management vendor Hyperion…”.

Apart from this “data quality” issue, the article does talk about some key BI truths:

(1) It’s about maximizing benefits, not minimizing costs. Price isn’t the number one criteria. BI is highly leveraged – it’s what unleashes the value of the millions of dollars that you’ve invested in your information systems. So a product that’s even just a little more powerful or easier to use is worth the extra cost.

(2) The benefits and use of BI are hard to predict in advance (“you don’t know what you don’t know”) — so choosing a vendor with a broad set of functionality can be a smart move.

(3) User adoption is key. Just implementing BI provides zero value — it has to be used, and has to result in changes to the business, and ease of use is necessary for wide usage (although not sufficient — training and change management skills are also essential)

Will there be a ”domino effect”?

A commentary in Business Week sums up the “domino effect” opinion (more prevalent among financial analysts than industry analysts) that the deal will lead to the purchase of the remaining large independent players. Here are some of the key quotes:

“The question is which makers of so-called business-intelligence software… will be acquired next.”

“The dam has burst in this sector,” says Murray Beach, president of Boston Corporate Finance, an investment bank that advises technology companies. “Business Objects can’t be far behind”.

“Bankers and analysts expect business-intelligence software makers Business Objects (BOBJ) and Cognos (COGN) to be acquired this year.”

Other commentators have taken this a step further, and talked about the end of independent BI as a category. E.g. Tony Baer of OnStrategies Perspectives:

“…BI has little reason to remain a standalone market.”  (See the excellent serious about consulting blog for a direct reply to this)

As Dave Kellogg, former Marketing VP of Business Obje ts, has noted in his blog, the business intelligence industry has been remarkably unconsolidated up until now. How long can this last? And if companies like Business Objects and Cognos are purchased by bigger players, does that also imply that there is no longer a future for independent BI?

Is independent BI no longer needed?

There are three main schools of thought that say that it has outlived its usefulness:

1. BI should be integrated into business applications (the Oracle/SAP argument). People want to access information as a seamless part of their operational applications. Vendors such as Oracle and SAP offer BI tools and data warehousing environments preconfigured for their applications.

2. BI should be integrated with standard user productivity tools (the Microsoft argument). People want to access information directly from Microsoft Office. Excel is arguably the most widely-used BI tool today, and Microsoft aims to provide a robust set of BI tools across their Office suite and applications.

3. BI should be integrated with search (the Google argument). Most corporate information is stored in unstructured format, and surveys show that users would like to be able to access their BI reports using the same easy-to-use search interfaces. Google has deals with all the major BI vendors, has purchased www.gapminder.org and is making some moves in the mobile consumer BI market.

Does this combination of pressures spell doom for the independent vendors? Not necessarily. 

• First, note that there is clearly some truth to each of these scenarios, but they are mutually exclusive — i.e. if BI is a standard part of Office, then it cannot also be a standard part of your SAP environment. This already implies that there’s room for independent players who can blend the best of each approach.
• Second, healthy BI market growth means that success for one set of vendors doesn’t have to imply failure for the others: everybody can expand as the “BI balloon” grows.

Is there a future for Independent BI?

The real question is “will there continue to be a competitive advantage for independent BI tools in the future?” I believe the answer is yes, for the following reasons:

Variety is the spice of life. Organizations will always need to access and analyze information from many different systems.

• Across the organization. Even in the rare cases where an organization has a single “standard” for their applications, much of the information that executives care about comes from a variety of smaller operational systems. And this situation is not going to go away. The search for competitive advantage through new and innovative systems (e.g. RFID…) means that a single source of data for the entire organization is an unreachable nirvana (if you ever get close to having a true standard, I promise you’ll merge with another organization and have to start again!). In addition, companies are reluctant to have only one strategic supplier and lose all bargaining power, and the vendors do everything they can to remain in accounts. One of the reasons Oracle gave for buying Hyperion was its strategic position in SAP accounts — i.e. to make it harder for SAP customers to completely standardize on their main vendor.

• Across the ecosystem. Today’s organizations are part of complex chain running from raw materials to the final customer. It’s rare to find an organization that doesn’t outsource part of their activities to specialists. You share operational information with your largest customers on a daily basis (although sadly often only via phone or fax). Even if your operational systems are with one particular vendor, you’ll need to be able to access information from across the ecosystem to get a true view of your operational activities.
• Across the industry. Business executives increasingly need external information as part of their BI environments, including industry statistics such as market share, customer satisfaction, and benchmarks against their competitors.
• Data integration isn’t enough. Data integration technology alone cannot solve this problem. There will always be some data sources that require too much time or cost to integrate into the data warehousing environment, and users will always need to integrate and analyze information from multiple sources on the fly.

Vested interests are important. There is no technological reason why Oracle cannot continue to provide excellent support for, say, Hyperion in SAP environments. In practice however, Oracle is in fierce competition with SAP and is unlikely to act with strict neutrality, particularly with regard to embedding BI into the operational workflows of SAP. The vendors protest that they are open and will do what’s best for their customers, but there’s a reason that sports teams and referees are discouraged from betting on the matches they play in.

Focus leads to expertise. Ten years ago, the benefits of buying a BI tool from your database vendor were just as obvious as the benefits of buying one from your application vendor today, but that didn’t stop the success of independent BI vendors. Focus comes down to what one cares about. Independent BI vendors care about their customers’ BI success. Oracle cares about beating SAP and selling more applications and databases. So far, the history of the industry indicates that vendors focused on BI are likely to provide a more innovative, customer-focused solution than those that view BI primarily as a means of selling their platform, databases, applications, or middleware.

BI is highly lev
eraged. BI is the final layer between business people and the millions of dollars you have invested in your information systems. A small difference in BI effectiveness can have a large effect on the overall return on IT investment.

Who’s the best customer for independent BI?

Organizations that:

• Have a combination of Oracle, SAP, Microsoft, and other domain-specific applications, and use MS Office.
• Are in fast-moving industries, may need to purchase or merge with another company one day, or otherwise need flexibility in their systems.
• Work closely with their ecosystem of partners, would like to share information with them.
• Use external information for benchmarking and need to share information with market and regulatory authorities.
• Care about using information to help turn strategy into execution at all levels of the organization.

In other words, just about everyone…

Maybe Oracle, SAP, and Microsoft will deliver on their promises this time. Maybe “good enough” is as much as most people need. Maybe Google will upset the whole industry. But I’m hoping that BI is strategic enough for independent BI to live up to its theoretical potential.

Finally, will Business Objects or Cognos get purchased? 

Who knows. Both are a public companies and the executive teams hold only a minority of the stock, so ultimately the potential suitor and the shareholders would decide. But any purchaser would lose a large part of the value of the company overnight, since it would no longer be independent, with all the advantages above.

Intelligent Enterprise: “Ultimately, Oracle caught on to the unusual volume of requests…. Oracle says an investigation into huge traffic spikes on its Customer Connection servers…”

Oracle’s complaint document: “In late November 2006, there occurred unusually heavy download activity on Oracle’s password-protected customer support website…”

“More than 10,000 illicit downloads from Customer Connection between September 2006 and January 2007, with indications that this number may go significantly higher if traced further back in time.”

In other words, it seems Oracle only noticed the problem because of network spikes, not because of any regular proactive BI analysis of support usage, and BI was only used after they noticed a problem.

As I noted in a previous post, BI is unfortunately often considered an afterthought of operational systems. It should be considered an integral part of enterprise security, with ongoing, regular analysis of all aspects of network behaviour.

Less change than you might expect

It doesn’t change the BI market share picture much, based on the latest IDC numbers, from 2005 (2006 figures presumably due out soon, not expecting any huge changes in the relative positions). Hyperion is currently #5, and Oracle is #6, and combined they would become #4, ahead of Microsoft but behind Business Objects, SAS, and Cognos.

EPM: Back to the future

A decade ago, Oracle Express / Oracle Financial Analyzer was the only real competitor to Hyperion Essbase for financial planning and budgeting. It was powerful but aging technology even before Oracle purchased it from IRI in 1995, and they stopped updating it in 2002. Oracle proposed Enterprise Planning and Budgeting (EPB) in 2004, but with completely different underlying technology. Customers howled, and many of them presumably turned to solutions like Essbase.

So In the financial planning and budgeting area, purchasing Hyperion just puts Oracle back to it might have been if it had provided a credible upgrade at the time — except, of course, there’s now clear product overlap with Express/EPB.

Ironically though, Essbase is now an aging solution itself, and will be inevitably perceived as less independent after the acquisition. And the market is moving to support the “CFO 2.0″, with fast innovation and opportunities for other vendors (e.g. Business Objects) to explain the value they can bring to the performance management/planning space.

Core BI: Ouch

Things don’t look good for Oracle/Hyperion core BI customers. Oracle already had lots of overlapping solutions, and this makes it much, much worse. It seems clear that the Oracle BI tools are to be retained, rather than Hyperion’s. Larry essentially justified the Siebel purchase by saying that Siebel Analytics was the “crown jewel of the acquisition“, and in 2Q of this year, Larry went on to say:

“In our business intelligence area, that area is really on fire for us. If you recall, we bought the Siebel analytics product, and that has become our base platform”

So what happens to the ex-Brio products that are now inconveniently integrated with the rest of Hyperions solutions? I’m sure Oracle will do a credible job trying to keep customers, but there’s only so much they can do:

• Existing customer deployments will be stalled trying to figure out which technology to invest in
• Innovation will be stalled while the various R&D teams reorganize
• Lots of Hyperion employees will decide that they don’t want to work for Oracle, and move to the remaining independents or smaller startups. The headhunters are surely speed-dialing as I write this…

Stuck with their “natural share”

Oracle has been claiming to be “serious about BI” for at least 15 years, and for 15 years they haven’t been able to break out of their “natural market share”, consisting of Oracle customers that aren’t that serious about BI, and who don’t evaluate the more powerful independent tools. I don’t believe that this acquisition will fundamentally change the equation.

Independent BI is here to stay — and thrive

There’s a clear value proposition for vendors who are focused only on BI and independent of databases, applications, middlewares, etc. Oracle made it clear that this is about them and SAP, not the customer needs — maybe it will help a few more companies choose independent companies that have their BI interests at heart.

There’s now one fewer vendor that qualifies, and the remaining independents can presumably look forward to getting some of the business from a potentially (probably?) dead-ended Brio customer base.

Good luck, Howard and Frank!

After jumping ship from Gartner to Hyperion, Howard Dresner and Frank Buytendijk will now be Oracle employees. I’m looking forward to reading their reactions…