Here’s a video from this years SAPPHIRE NOW in Madrid, featuring Mimi Speir, Steve Lucas, and Andrew Murray giving an overview of SAP BusinessObjects mobile directions:

Highlights include:

• New SAP BusinessObjects experience mobile application, based on the Exploration Views functionality from the BusinessObjects innovation center, with demo data from Experience.SAP.com
• Easily take a mobile analysis and create a StreamWork activity for collaborative decision-making
• Easily “mobilizing” your existing BusinessObjects reports
• Support for each level of the “mobile needs hierarchy”, including fully customized mobile applications

• Demonstration of integration with RightHemisphere, a recent SAP acquisition that allows companies to synchronize visual and business data, combining a camera and schematic view:

Overlaying colors by temperature from data in the ERP system:

Finding a required part replacement using geolocation data:

I was honored to speak at this years UK & Ireland SAP User Group Conference held in Birmingham, UK. The first main day saw great keynotes from the new SAP UK Managing Director, Steve Winter; Analyst Ray Wang; SAP’s CIO, Oliver Bussmann; and Olympic Sprinter Steve Cram. You can read a full review by Christoph Ziedler on SAP.info, and see the various videos on YouTube starting with the introduction by Alan Bowling, Chairman of UKISUG.

I gave the keynote on the second main day. Here’s the video taken by the organizers, including the slides:

You can download my slides in pdf format or in the original PowerPoint format.

Here’s a selection of photos I took during the sessions:

And there are many, many more photos on the UKISUG Flickr Stream.

When was the last time you had a roll of film developed?

If you’re reading this, you’re no technology luddite, so it was probably already many years ago. Film – a technology that had been slowly improved and perfected over 200 years  – was replaced in less than a decade by a much faster, cheaper, and more convenient technology approach.

In an earlier post, I outlined why I think we’re on the brink of a real revolution in business analytics infrastructures. This post draws out some of the parallels with the shift from analog to digital image processing, and the effects it had on the industry as a whole.

Film Cameras Compared to Today’s Data Warehouses

Data warehousing today has a lot in common with the first photos I took fifteen years ago with my then-brand-new Canon Elan II camera:

High Costs

Buying film and getting pictures processed got very expensive very quickly. Only a tiny minority of professionals could take as many shots as they wanted – the rest of us had to be choosy about our shots.

Today’s data warehousing is complex and expensive, and organizations have to ruthlessly prioritize which projects will be undertaken. Meanwhile, other worthy projects have to wait, business users have to rely on gut feel.

Upfront Planning Required

In order to get the best results, you had to know in advance what kind of thing you were going to be taking shots of. Want to take a landscape shot? You needed 100 ISO and daylight film. An action shot in a hockey rink? 1600 ISO and tungsten lighting. A colorful tropical scene? High-saturation Fujifilm Velvia. A moody night shot of Paris? High-grain Kodak black-and-white xyz….  And the film only came in 24 or 36 pictures – you just changed your mind about what you wanted to take picture of? Tough.

Today’s data warehouses require you to decide in advance what data you’ll want to access later – and if you change your mind, you have to go back to the source data and reload it with different transformations.

Slow Feedback Loop

What you saw through the eye-piece was very different from the resulting picture. By the time you got the film back and realized that the exposure was wrong, or the framing a little off, there wasn’t that much you could do about it. The slow feedback loop meant that lots of good pictures were missed, and that you had to take more pictures than you needed. Want a great shot of a high-contrast night scene? You needed to take multiple long exposures (with the associated high costs) in order to have a hope of getting what you wanted.

By the time business people have had a chance to access the data, reports, and dashboards in your business intelligence solutions, you’ve already put in huge amounts of efforts – and the business has moved on, and now has different needs. To try to avoid this, you try to add all the data you might need into the data warehouse, but this increases the project cost and complexity.

Arcane Knowledge

With enough training, skilled photographers understand all the variables involved in getting the right picture, and were able to get more consistent, reliable results. But even they had to rely on specialized film processors to actually get the results.

Because of all the factors mentioned above, data warehousing today requires lots of arcane knowledge to be successful, and the people that know how to do it are in high demand and short supply.

Digital Cameras Compared to New Analytics Platforms Powered by In-Memory and Other Technologies*

I purchased one of the first consumer digital cameras, the Sony DSC-F1. It transformed the way I took pictures, just as in-memory computing is transforming analytics.

Cheap, Easy, Iterative Learning and Experimentation

The screen shows you exactly the shot that you’re going to take, and digital cameras don’t require film or processing, so the marginal cost of another picture is effectively zero. You can try things, and if they don’t work out, you can instantly adjust your approach and try again. Things like f-stops still make a difference to the end result, but you can learn how it works in an intuitive way, rather than having to wade through mathematics. Previously essentially accessories like light meters become unnecessary – you can get the effect you need through trial-and-error in the same time it took you to work out the light readings.  We are all better photographers now – because we can experiment, and throw away 90% of our pictures without worrying about it.

Once you have the row-level data in-memory, you can easily change your analytic view on the fly. You want to use a different attribute, rolled-up in a different way? Rather than having to reload and retransform the data, you can simply make a change to the metadata, and the users get the results they want. You can quickly and iteratively prototype your business intelligence solutions, rather than having to try to rigorously plan everything in advance. Faster, more convenient analytics results in better analytics.

Faster Innovation

The combination of photography and computing has transformed what you can do with photos. Here are just some of the things that were almost unimaginable with analog film:

• Different camera angles. You used to have to view everything through the eyepiece. Replacing it with a screen made it easy to get a new view on what you were looking at
• Taking a picture of something that happened in the past. This seems like science fiction, but several digital cameras use a buffer to store the last few seconds of whatever you’re looking at, so even if pressed the shutter too late when your daughter scored the winning goal, you can back up to
• High dynamic range. The human eye sees more shades than any current camera. The latest cameras automatically take several shots and combine them to create a full range of shades, and you can choose the exposure afterwards.
• Selective focus. Today, some celebrity photographers only take pictures fully in-focus, and rely on Photoshop to introduce focus wherever they need it. And a brand-new camera promises to use some cool multi-lens technology and digital processing to let the rest of us do this very easily.

We’re just at the very start of what we’ll be able to do with in-memory systems. For example, it seems that in-memory column stores are well-adapted to extending enterprise analytics to unstructured data, real-time data, social data, etc — things we’ve been struggling with using traditional data warehouse approaches.

Integration With Other Systems

My main “camera” is now my iPhone, and has become a “feature” in a larger system. The image quality of the iPhone camera isn’t great, but combined with its convenience, the flexibility of the apps, HD video, and the ability to instantly share the results, it’s replaced my larger Canon Digital EOS camera for most things.

In-memory analytics isn’t just about analytics – these technologies will be integrated directly into operational systems. There will be “one version of the truth”, because we’re doing everything with one set of data…

Morals of the story

• Digital photography transformed an industry by eliminating obsolete layers. In-memory analytics and related technologies* will do the same.
• The change from analog to digital photography didn’t happen overnight. The digital cameras were relatively expensive compared to film, and some kinds of pictures made more sense, and in particular it took a time for the new digital cameras to rival the effective picture resolution of larger-format films. “Old-style” data warehousing won’t vanish overnight, but it will inevitably be relegated to particular types of tasks as in-memory analytics becomes more robust and takes on larger volumes of data.
• Today, many BI projects end up in failure, just like most of your old photographs. In-memory analytics will improve the quality and success of analytics projects.
• Some people jumped on the early limitations of digital cameras and insisted that the answer was to tweak the existing methods (buy scanners, etc.) – which missed the bigger picture. Today, some people try to insist that in-memory is “just a memory cache”, and that incremental technologies like Flash Disk / SSDs are the answer. Don’t be in denial.
• If your job relies on your existing data warehousing skills, better get used to the new world, or move to another role…

For more about SAP HANA and in-memory technology, visit http://experiencesaphana.com. And if you are interested in photography, you can find my photos here: http://blog.timoelliott.com (daily updates) and http://timoelliott.com/personal

_____________________

*Updated from original post — although I say “in-memory”, it’s really about a collection of various technologies including in-memory, massively parallel hardware, column stores, and in-database analytics – please see the link at the start of the post for more details.

Many thanks to William Gardella and others for the content below:

Traditional enterprise data warehousing and Hadoop/Big Data are like apples and oranges – the well-known and trusted approach being challenged by a zesty newcomer (sweet oranges were introduced to Europe sometime in the 16th century). Is there room for both? How will these two very different approaches co-exist?

This post is an attempt to summarize the current state of play with Hadoop, “Big Data” and Enterprise BI, and what it means to existing users of enterprise business intelligence. See the list of articles at the end of the post for more detailed materials.

What is Hadoop?

Hadoop is open-source software that enables reliable, scalable, distributed computing on clusters of inexpensive servers. It is:

• Reliable: The software is fault tolerant, it expects and handles hardware and software failures
• Scalable: Designed for massive scale of processors, memory, and local attached storage
• Distributed: Handles replication. Offers massively parallel programming model, MapReduce

Hadoop is designed to process terabytes and even petabytes of unstructured and structured data. It breaks large workloads into smaller data blocks that are distributed across a cluster of commodity hardware for faster processing. And it’s part of a larger framework of related technologies:

• HDFS: Hadoop Distributed File System
• HBase: Column oriented, non-relational, schema-less, distributed database modeled after Google’s BigTable. Promises “Random, real-time read/write access to Big Data”
• Hive: Data warehouse system that provides SQL interface. Data structure can be projected ad hoc onto unstructured underlying data
• Pig: A platform for manipulating and analyzing large data sets. High level language for analysts
• ZooKeeper: a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services

Image: William Gardella

Are Companies Adopting Hadoop?

Yes. According to a recent Ventana survey:

• More than one-half (54%) of organizations surveyed are using or considering Hadoop for large-scale data processing needs
• More than twice as many Hadoop users report being able to create new products and services and enjoy costs savings beyond those using other platforms; over 82% benefit from faster analyses and better utilization of computing resources
• 87% of Hadoop users are performing or planning new types of analyses with large scale data
• 94% of Hadoop users perform analytics on large volumes of data not possible before; 88% analyze data in greater detail; while 82% can now retain more of their data
• Organizations use Hadoop in particular to work with unstructured data such as logs and event data (63%)
• More than two-thirds of Hadoop users perform advanced analysis — data mining or algorithm development and testing

How is it Being Use in Relation to Traditional BI and EDW?

Currently, Hadoop has carved out a clear niche next to conventional systems. Hadoop is good at handling batch processing of large sets of unstructured data, reliably, and at low cost. It does, however, require scarce engineering expertise, real-time analysis is challenging, and it much less mature than traditional approaches. As a result, Hadoop is not typically being used for analyzing conventional structured data such as transaction data, customer information and call records, where traditional RDBMS tools are still better adapted:

“Hadoop is real, but it’s still quite immature. On the “real” side, Hadoop has already been adopted by many companies for extremely scalable analytics in the cloud. On the “immature” side, Hadoop is not ready for broader deployment in enterprise data analytics environments…” James Kobelius, Forrester Research.

To considerably over-simplify: if we consider what’s called the 3 ‘V’s of the data challenge: “Volume, Velocity, and Variety” (and there’s a fourth, Validity), then traditional data warehousing is great at Volume and Velocity (especially with the new analytic architectures), while Hadoop is good at Volume and Variety.

Today, Hadoop is being used as a:

• Staging layer: The most common use of Hadoop in enterprise environments is as “Hadoop ETL” — preprocessing, filtering, and transforming vast quantities of semi-structured and unstructured data for loading into a data warehouse.
• Event analytics layer: large-scale log processing of event data: call records, behavioral analysis, social network analysis, clickstream data, etc.
• Content analytics layer: next-best action, customer experience optimization, social media analytics. MapReduce provides the abstraction layer for integrating content analytics with more traditional forms of advanced analysis.

Most existing vendors in the data warehousing space have announced integrations between their products and Hadoop/MapReduce, or their intention to provide them – for example, SAP has announced that they intend to implement MapReduce in the next version of Sybase IQ.

What Does The Future Look Like?

It’s clear that Hadoop will become a key part of future enterprise data warehouse architectures:

“The bottom line is that Hadoop is the future of the cloud EDW, and its footprint in companies’ core EDW architectures is likely to keep growing throughout this decade. “ James Kobelius, Forrester Research

But (despite some of the almost religious fervor of its backers) Hadoop is unlikely to supplant the role of traditional data warehouse and business intelligence:

“There are places for the traditional things associated with high-quality, high-reliability data in data warehouses, and then there’s the other thing that gets us to the extreme edge when we want to look at data in the raw form”  Yvonne Genovese, Gartner Inc.

Companies will continue to use conventional BI for mainstream business users to do ad hoc queries and reports, but they will supplement that effort with a big-data analytics environment optimized to handle a torrent of unstructured data – which, of course, has been part of the goal of enterprise data warehousing for a long time.

Hadoop is particularly useful when:

• Complex information processing is needed
• Unstructured data needs to be turned into structured data
• Queries can’t be reasonably expressed using SQL
• Heavily recursive algorithms
• Complex but parallelizable algorithms needed, such as geo-spatial analysis or genome sequencing
• Machine learning
• Data sets are too large to fit into database RAM, discs, or require too many cores (10’s of TB up to PB)
• Data value does not justify expense of constant real-time availability, such as archives or special interest info, which can be moved to Hadoop and remain available at lower cost
• Results are not needed in real time
• Fault tolerance is critical
• Significant custom coding would be required to handle job scheduling

Does Hadoop and Big Data Solve All Our Data Problems?

Hadoop provides a new, complementary approach to traditional data warehousing that helps deliver on some of the most difficult challenges of enterprise data warehouses. Of course, it’s not a panacea, but by making it easier to gather and analyze data, it may help move the spotlight away from the technology towards the more important limitations on today’s business intelligence efforts: information culture and the limited ability of many people to actually use information to make the right decisions.

References / Suggested Reading

• Hadoop Goes Mainstream for Big BI Tasks
• Hadoop: Is It Soup Yet?
• Hadoop: What Is It Good For? Absolutely . . . Something
• Hadoop: What Are These Big Bad Insights That Need All This Nouveau Stuff?
• Hadoop: Future Of Enterprise Data Warehousing? Are You Kidding?
• Hadoop: When Will The Inevitable Backlash Begin?
• Hadoop finds niche alongside conventional database systems
• ‘Big data’ analytics fulfilling the promise of predictive BI
• Big data: The next frontier for innovation, competition, and productivity

It’s an unspoken assumption that mobile BI extends normal BI: almost all the coverage so far emphasizes the usefulness of these devices for people out of the office: on the road, or on the factory floor – and it’s hard not to agree that there is a great opportunity to bring business intelligence to new classes of users. In particular, executives spend their lives running around visiting different departments and divisions, asking people to justify (and improve) their performance. Having the dashboards at their fingertips, as they ask the questions, is an extremely powerful tool that makes the managing process more efficient (aka “if you’re finished arguing your opinions, I actually have some data…”)

But I think “mobile BI” goes further than that, and will increasingly replace existing full-client BI systems. Today’s mobile devices aren’t just small enough to stick in your pocket, they also tend to use state-of-the-art, multi-touch interfaces.  Just like Tom Cruise in Minority Report, it’s simply more intuitive and easy to access  information using your fingertips than it is a mouse. Increasingly, I find myself reaching for my iPad to access data, rather than my laptop, even when it’s right in front of me.

Today’s mobile devices are simply better for analysis, and their children will replace BI on PCs, not just augment it. Looking to the future, the lines between mobile and traditional devices is blurring fast: tablets are becoming more powerful, and supporting “traditional” operating systems like Windows, and laptops are starting to come installed with multi-touch touchpads, GPS and 3G connections. “Mobile” will no longer be a separate environment, but a seamless part of a normal rollout.

One factor, however, may delay the process. For many years, vendors have had the luxury of fairly simply platform choices in the enterprise world: Windows or a browser as the front end, Windows/Unix/Linux as the back end. But there’s been an explosion of different mobile device operating systems and multi-touch interfaces, with no end to the confusion in sight. HTML 5 will almost certainly help, but there’s a long way to go before it’s strong enough to replace existing choices for intuitive, device-optimized, multi-touch interfaces. Multiple choices means extra work for the vendors, and slower deployment/adoption of the new interfaces in enterprise environments.

SAP Social Intelligence, born as the social network analytics prototype from the SAP BusinessObjects innovation center (now SAP Research Prototypes), is now fully launched and available as an integral part of the SAP StreamWork collaborative decision solution.

With so much going on with various launches including BusinessObjects version 4.0 and HANA, you may have missed the launch of the Enterprise Edition of StreamWork, but it was a key step for integration with the rest of enterprise information infrastructure, and SAP has announced that StreamWork is a central part of the ongoing collaboration strategy — not just of business analytics, but also SAP’s core applications like CRM and PLM.

The project manager, Sharon Haver, has just posted a brand-new video that explains social intelligence using funky blue people (including the key employees that work on the StreamWork product) tied together with rope – it looks like they had a lot of fun making it…

Enjoy.

Let’s take a look at an analogy from the history of computing. The first ever computer used for commercial business applications was created in 1951, called “Lyons Electronic Office” (LEO). J. Lyons and Co. was the Starbucks of its day, with high-volume tea rooms open 24 hours a day in key locations across the United Kingdom.

Lyons used LEO to solve a classic business analytics problem that organizations still struggle with today. They created a “Bakery Valuation application” that let the company optimize profitability and minimize waste by calculating exactly how many perishable buns and tea-cakes should be produced for the next day, based on the latest purchase data available. The very first application on the very first commercial computer was already all about business analytics.

LEO was the Exadata of its era – it was the biggest and best data-crunching machine available, with more than double the memory of its nearest rival, the Colossus. Sixty-four 5ft-long mercury tubes, each weighing half a ton, were used to provide a massive 8.75 Kb of memory (i.e. one hundred-thousandth of a today’s entry-level iPhone).

LEO provided breakthrough performance. It could calculate employee pay in 1.5 seconds, replacing skilled clerks that took 8 minutes. But LEO was already a dinosaur, about to be replaced by a completely new technology.

Leo used over 6,000 vacuum tubes to carry out calculations. They worked, but they were complex, large, slow, fragile, expensive, and generated massive amounts of waste heat and noise. Engineers could detect problems simply by listening to the cacophony of buzzes and clicks generated by the machine.

Then a technology breakthrough came along: the transistor. Invented in 1947, they were much simpler, much smaller, much cheaper, more reliable, and much, much faster than vacuum tubes. The first transistor-based computers appeared in 1953, radically changing what was possible with electronics, and rapidly consigned LEO to the dustbin of history.

And transistors were just the start of the revolution. As technology improved and miniaturized, integrated circuits were created to pack millions of transistors onto a single chip, enabling previously unthinkable possibilities (try to imagine a vacuum-powered iPad!).

I believe that we are rapidly moving from the “vacuum tube era” of BI and data warehousing to the “transistor era”. Today’s best-practice BI architectures are rapidly becoming obsolete, and we can already start imagining what the new “integrated circuit” opportunities of the future might look like.

The last decade of “traditional best-practice” BI has been based on the following architecture:

1. We start with business applications that gather the data we would like to analyze.
2. We can’t do more than basic reporting against this data without slowing down the system, so we create a copy that’s typically called an “operational data store” or ODS.
3. The ODS doesn’t store history, we want to analyze data from multiple systems, and the data is incompatible or incomplete, so we use ETL (extraction, transformation, and loading) technology to load data into database structures optimized for business intelligence – a data mart or data warehouse.
4. Businesses want to store lots of information. To provide acceptable query times, the data warehouse must be optimized by the addition of specialized data structures, database indexes, and aggregate tables – all of which add to the size and complexity of the data warehouse.
5. Business intelligence tools are made available to business people to access and display the information they are interested in. To provide better interactivity, an additional data cache is often created for a particular report or cube.
6. Because this architecture is slow and unwieldy, organizations often create extra data marts for a particular business need.

The result is a vacuum tube: it works, and it’s the best alternative we have right now, but it’s slow, complex, and expensive.

Faced with these frustrations, several technologies have been used over the years to increase business intelligence speed and flexibility. Each has provided valuable progress, but has some downside that prevented it being used on a more general basis. A tipping point has arrived, however, and a combination of these approaches holds out the promise of a radically simpler BI architecture.

The most important is “in-memory processing”. All computer processing has always happened in live memory, but up until now, there have been severe limitations on how much data could be stored, and so all data has first to be retrieved from disk storage before it can be processed.

Over time, memory processing capabilities has expanded exponentially, in line with Moore’s Law, doubling every few years. But disk access speeds have been limited by real-world aerodynamics, and have increased only by 13x or so over the last fifty years. The result has been an ever-widening gulf between the speed of processing data and retrieving it from disk. Today, it can be up to a million times slower to get data from disk than from live memory.

This leads to tough architecture choices. One way of imagining the consequences is to compare it to a chef cooking a meal. If — like on the TV cooking shows — all the ingredients are already prepared and sitting on the counter-top, it’s very quick and easy to create the meal. This is the equivalent of “in-memory processing” once all the required data is available.

But imagine now that the chef doesn’t already have the ingredients ready. Given the slow relative speed of disk access, it’s as if the closest supermarket was on the planet Mars, and the ingredients had to travel months by rocket before each and every meal.

Database vendors have taken every approach possible to increase disk access speeds, for example by predicting what data is most likely to be needed and caching it in advance (the equivalent of pre-stocking a larder in the restaurant, full of the ingredients that are most requested). But the whole point of a data warehouse is to be able to ask any question –  the equivalent of being able to order any meal in the restaurant — and so you have to go back to the supermarket on Mars on a regular basis.

Up until recently, it’s simply been too expensive to store data anywhere other than disk. But the price of memory has plummeted over the last two decades, and 64 bit addressing has radically increased how easy it is to access. Just ten years ago (when we first defined the current BI best practices) the price of one megabyte of live memory was around one dollar. Now it’s over a hundred times less: below one cent, and still falling fast. This is equivalent to something shrinking from the size of the Statue of Liberty down to a Chihuahua: it would be strange indeed if this didn’t have an impact on how we create our BI architectures.

If the whole data warehouse could be stored in-memory, we could make the whole system much faster and much simpler – and eliminate the need for disk-based storage altogether. We’d no longer have need for database optimizations like aggregates and indexes – and eliminating these simplifies the data loading process, and allows us to store more data in that limited, valuable memory space.

But in-memory processing alone only gets you so far – to get the full value of in-memory processing, we want to pack in as much data as possible, and to do that, we can turn to a complementary technology: column data stores.

Today’s relational databases are row-based: each new set of data is written into the next-available memory space. This is fast, which is essential for high-volume transactional applications writing to slow disks. But there are downsides for storing analytic data in a row-based structure, in terms of storage efficiency and query speed.

Let’s use an analogy to illustrate the difference between the systems: I employ a “row-based” filing system at home. I open each day’s mail, take a quick look, and then put it on top of a big pile in the corner of my bedroom.  At one level, it’s an extremely efficient system: it’s very fast to “write” to the database, and if I want to find all the papers I received on a particular date (a “transaction”), I can find it pretty quickly.

But if I want to do some “analysis”, such as finding my last five bank statements, it’s slow and painful: I have to systematically go through the whole pile (a “full table scan”). I could make things faster by, say, adding yellow post-it notes to the corners of bank statements, so I can go straight to that type of document (a “database index”), but that would create extra work and complicate the system.

My (far more organized) wife uses a “column-based” filing system. When she receives her mail, she takes the time to sort out the documents and allocate them to separate folders. It’s initially slower to store information, but it’s much, much faster when she wants to find all her bank statements.

Column databases store data more efficiently, and allow greater compression, because you store similar types of information together. For example, I get paid the same amount each month, so rather than storing the same pay slip twelve times in the file, I could simply store it once, with a link to each month, and add an exception for my annual bonus. The result is that you can store between ten and a hundred times more data in the same physical space, shrinking the data warehouse back down to a size similar to the raw data used to create it (see diagram below). This in turn reduces the amount of memory you need to scan for each query and increases query processing speed.

Commercial column databases such as Sybase IQ have been around for a long time, and have proved their value, particularly in very high-volume data warehousing environments. But the extra up-front loading time, compared with the slow write-time to disks, has limited their use in the market.

Now let’s imagine combing the two technologies, with an in-memory, column database. Because it’s compact, we can now store the entire data warehouse in memory. And because it’s fast, loading times are no longer a problem. We now have the best of both worlds without the downsides of each: the equivalent of being able to store the whole supermarket in the chef’s kitchen.

But we haven’t finished yet. We can bring three other data warehouse optimization techniques into the mix: analytic appliances, in-database algorithms, and incremental loading.

By building a massively-parallel machine specifically for data warehousing, we can again radically increase the speed we can access and manipulate data. In particular, column databases are very well adapted to parallelization: because each column is stored in a separate area of memory, aggregates can be efficiently handed off to a separate processor, or even partitioned across several processors.

To go back to the filing analogy: it’s the equivalent of my wife and I both working on our finances. If we had to work off the same pile of “row-based” documents, we’d constantly be in each other’s way. But with separate “column-based” folders, my wife can look through the documents of one bank while I look through another, or we can split the folder in two and each look at one half of them (“partitioning”).

We can also radically improve query speed by doing as much work as possible directly in the database. Today, if you want to do some statistical analysis, you typically have to do a query on the data warehouse, extract a large set of data to a separate statistics tool, create a predictive model, and then apply that to your data.

To take an overly simplistic example: imagine you wanted to rank one million customers by their cumulative revenue. if ranking is available directly in the database engine, you only have to extract one line of data – the result – rather than a million rows.

Of course, by having all the required data in-memory, and with the support of massively parallel processing, we can imagine far more sophisticated operations than just ranking. For example, accurate profitability and costing models can require huge amounts of processing. Being able to do it directly in the in-memory database would take a fraction of the time it typically does today.

Using a separate appliance also allows us to simplify our architecture: we can add a BI server in the same machine, without the need for a separate local calculation engine.

Next, instead of finding and updating data values in our data warehouse, we’ll just incrementally replicate new data from the operational system as soon as it’s available, and add a time-stamp. For each query, we simply ignore any out-of-date data. The loading process becomes much simpler and faster: we can just add new data as it arrives without having to look at what’s already in the data warehouse. This also provides a big auditing and compliance advantage: we can easily recreate the state of the data warehouse at any point in time.

Since we’re only loading new data, and we can do so in real-time, we no longer need an operational data store, and can eliminate it from our architecture.

It’s worth noting at this point the virtuous circle created by these different technology advances working together. Different vendors in the industry typically concentrate on combining one or two approaches. Each provides an improvement, but combining them all is the real opportunity: together, they provide all the upside benefits while mitigating the downsides of each technology – and lead to the real tipping point, where you can realistically give up the disk-based storage.

Thanks to in-memory processing, column databases, hardware optimization, in-database calculations and incremental loading, we’re left with the “transistor” version of business analytics. It does the same thing as the previous architecture, but it’s radically simpler, faster and with a more powerful architecture.

This diagram represents the “new best practice” possibilities of the new generation of analytic platforms. But we can go even further, and start thinking about the “transistor” phase of analytics – what new things might be possible in the future?

If we add ACID compliance (a set of properties required for reliable processing of operational data), and the option of using row and/or column storage in the same database, then we could use the same platform for both transactional and analytic processing. Backup storage would be provided by solid-state devices, so you could always recreate your in-memory environment.

This would have several advantages. First, we’ve talked a lot about having a “single source of the truth” in the last decades of business intelligence – if the transactions and the analytics are happening off the same set of data, then we can get a lot closer to our goal.

Second, we get closer to “actionable intelligence”, the ability to provide people people with analytic information as a seamless part of their operational activities, in time to make a difference (e.g. predict that the store will be out of stock tomorrow, and arrange a new delivery, rather than just telling us that the store ran out of stock yesterday).

Third, the new architecture is surprisingly well adapted to real-world business applications, which typically require much more than simple read-and-write operations. Application designers would no longer have to create complex logic above the database layer to do things like budget allocations or supply-chain optimization – these could use the superior analytic power of the analytic engine directly within the new platform.

We could then extend this architecture – putting it in the cloud would make mobile business intelligence, extranets, and data collaboration (both inside the organization and across the “business web”) easier and simpler.

There are also enterprise information management advantages. For example, one common business frustration with BI has been how hard it is to compare corporate data stored in the data warehouse with personal or external data. Today, business people have to extract a large download from the data warehouse, and manually combine it with other data using Excel. This adds extra complexity and work, and leads to information governance issues.

With the new architecture, there’s no longer any need for painful staged uploads to the data warehouse – we can create a “sandbox” area in the analytic platform, let people upload their data in simple row format, and combine and analyze as they wish, using the same, standard, secure corporate infrastructure.

It also turns out that column databases do a good job of storing text data for easy search and retrieval, and other forms of data and algorithms (XML, hadoop) could potentially use the same infrastructure.

There’s one key thing to note at this point: the diagram seems to imply that “data warehousing” is no longer necessary. But nothing could be further from the truth. Reality is, and always will be, messy. The core need to transform, integrate, and model data, across a wide variety of different sources, is as important as ever.

Data integration, metadata, and data quality issues are business problems that don’t magically disappear with a new technical infrastructure. But we can use the power of the in-memory calculations to do enterprise information management in real-time, rather than batch. It becomes more practical to integrate data on the fly (“virtual data warehousing), and we can take the data quality algorithms we use today (fuzzy matching, comparisons to master data, etc.), and execute them as transactions happen. We can store both the raw data, and the “validated” or “corrected” values. This allows us to flexibly decide, at query time, how much reliability we need for a particular analytic need.

In conclusion

The last decade of BI best practice data warehouse architectures is rapidly becoming obsolete, as a combination of existing technologies comes together to provide a tipping point. Every vendor in the market includes some combination of in-memory, column databases, appliances, in-db calculations, map-reduce-type algorithms, and combining operations and analytics in their vision. The obvious conclusion is that the real vision is to use all of these technologies at the same time.

The new analytic platforms won’t magically cure everything that plagues today’s BI projects, but will lead to two big changes:

1. It gives us more simplicity and flexibility, to be able to implement business analytics fast, without having to spend most of our time tuning the infrastructure for performance.
2. It gives us the power of big data and real real-time performance, combining the best of operations and analytics to create new applications that simply aren’t feasible today.

Other resources:

• Gartner: “Data Warehousing Reaching Its Most Significant Inflection Point Since Its Inception”
• IDC: “What do a cluster of corporate events tell us about the specialty data warehousing market?”
• Merv Adrian and Colin White: “Analytic Platforms: Beyond the Traditional Data Warehouse”
• Hasso Platner: “A common database approach for OLTP and OLAP using an in-memory database” and book: “In-Memory Data Management: An Inflection Point for Enterprise Applications”

I often mention in my presentations that “120% of us don’t understand statistics” – and here’s the proof: somebody marked my presentation as 7 out of 5 for content and style:

(many thanks, whoever you are!)

I’m still catching up from the SAP BusinessObjects user conference a few weeks ago. One of the sessions I attended was David Huber’s overview of operational business intelligence and the soon-to-be-launched SAP BusinessObjects Event Insight product.

He started the session with the results of a recent CIO.com survey of Fortune 500 CIOs (CIO Technology Priorities July 2009), showing that “Business Process Management” and “Business Intelligence” are the #2 and #3 priorities for 2010.

Bring these two trends together and you get the need for “operational BI”: real-time, actionable BI directly within your operational systems. SAP has now combined event processing with business process management and dashboards to create new operational BI capabilities.

The key new element of this system is SAP Event Insight. The product originally started as a prototype from the SAP Imagineering Team called Live Enterprise, with links to dashboards from the SAP BusinessObjects Innovation Center, using complex event processing (CEP) technology from a small startup called Coral8. Coral8 was then acquired by Aleri, who was then acquired by Sybase, who was then acquired by SAP – all in less than 18 months! – and the product is now close to “ramp up”. For more information about BI 4.0, check out the SAP Analytics microsite.

The goal of Event Insight is to provide real-time monitoring of business events:

• Detect meaningful business events and new opportunities, such as customer buying behavior, through historical and current events
• Send alerts to business users, letting them detect and react to business changes before adverse events happen
• Easily set up the agents, on multiple data sources, monitoring both structured and unstructured data, without programming

Event Insight uses technology that was first developed for high-velocity, high-transaction trades on Wall Street, but has now been extended and adapted for use in BI. The biggest difference is in the nature of the processing: compared to previous “monolithic”  CEP implementations, business intelligence typically requires more data sources, with processing shared among nodes on the network.

The new technology is tightly integrated with the SAP BusinessObjects semantic layer, letting you create universes and queries on top of event streams, linking them to the BI 4.0 alerting framework, and letting them be accessed from standard front end tools such as Web Intelligence, Crystal Reports and Dashboards (formerly known as Xcelsius).

So when might you need this technology? Here’s David’s comparison of traditional and operational BI approaches: unsurprisingly, operational BI is operational processes, where you need to take action on information in a short time.

Operational BI is a very horizontal technology that could be used in lots of different industries and applications. Examples given included:

• A supply chain example. The instant that the system detects a problem with an order, an alert is sent to customer care, who can proactively warn the customer, and mitigate any delay
• Oil and gas. Real-time pressure, temperature, and vibration readings are gathered and sent to a central dashboard, and any readings outside of defined limits are sent as alerts.
• Location-based processing. The event trigger could be somebody or something arriving at a particular location (leveraging the Sybase mobility technology)
• Monitoring customer sentiment. Look for keywords in a stream of social media using text analytics, and send alerts.

Demonstrations

The SAP Community Network site has some great demonstrations available — here are links to some of them. First, a demonstration of how the technology could be used by an Oil and Gas company:

Here’s a description of how operational BI can used for improving the number of “perfect orders”:

And here’s an associated demonstration of the perfect order dashboard:

Operational BI can be used for tracking and tracing defective products, including collaborative decision-making with StreamWork and SAP’s business process modeling.

And here’s the associated product defect demo, with social media monitoring and drill-down to SAP systems:

David mentioned that an industrial poultry operation was even interested in the technology – but declined to explain further!

It’s claimed that each individual agent install is quick and easy, with both SAP and non-SAP systems. As with traditional BI, the trickiest part is deciding the business process issues and trigger conditions that you want to measure. There are three types of user profile who would maintain and use the system:

• IT – somebody to install the system, and collect, process, and manage data
• BI power user – to set up the event semantics, types of event, thresholds, etc.
• Business user – consumes the data and alerts

Since some of the examples overlap with what you can do with traditional BI today, I asked David for his honest assessment of how applicable and practical this technology will be for “average BI-using organizations”, including the cost of buying, installing, and maintaining it in addition to existing systems. He replied that he believes that operational BI won’t be confined to just a few niches (finance, etc.), but will become a standard part of most large-organization BI deployments.

Complementary information:

• Check out the excellent SAP analytics site
• some publicly-available slides from an AFSUG event earlier this year