![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image.jpg?resize=690%2C359&ssl=1\" "\\"image\\"")
<\/p>\n
Lyons used LEO to solve a classic business analytics problem that organizations still struggle with today. They created a \u201cBakery Valuation application\u201d that let the company optimize profitability and minimize waste by calculating exactly how many perishable buns and tea-cakes should be produced for the next day, based on the latest purchase data available. The very first application on the very first commercial computer was already all about business analytics.<\/p>\n
LEO was the Exadata<\/a> of its era \u2013 it was the biggest and best valve-based data-crunching machine available, with more than double the memory of its earlier rival, the Colossus<\/a>. Sixty-four 5ft-long mercury tubes, each weighing half a ton, were used to provide a massive 8.75 Kb of memory (i.e. one hundred-thousandth of a today\u2019s entry-level iPhone).<\/p>\n LEO provided breakthrough performance. It could calculate employee pay in 1.5 seconds, replacing skilled clerks that took 8 minutes. But LEO was already a dinosaur, about to be replaced by a completely new technology.<\/p>\n Leo used over 6,000 vacuum tubes<\/a> to carry out calculations. They worked, but they were complex, large, slow, fragile, expensive, and generated massive amounts of waste heat and noise. Engineers could detect problems simply by listening to the cacophony of buzzes and clicks generated by the machine.<\/p>\n Then a technology breakthrough came along: the transistor<\/a>. Invented in 1947, they were much simpler, much smaller, much cheaper, more reliable, and much, much faster than vacuum tubes. The first transistor-based computers appeared in 1953, radically changing what was possible with electronics, and rapidly consigned LEO to the dustbin of history. And transistors were just the start of the revolution. As technology improved and miniaturized, integrated circuits were created to pack millions of transistors onto a single chip, enabling previously unthinkable possibilities (try to imagine a vacuum-powered iPad!).<\/p>\n I believe that we are rapidly moving from the \u201cvacuum tube era\u201d of BI and data warehousing to the \u201ctransistor era\u201d. Today\u2019s best-practice BI architectures are rapidly becoming obsolete, and we can already start imagining what the new \u201cintegrated circuit\u201d opportunities of the future might look like.<\/p>\n The last decade of \u201ctraditional best-practice\u201d BI has been based on the following architecture:<\/p>\n The result is a vacuum tube: it works, and it\u2019s the best alternative we have right now, but it\u2019s slow, complex, and expensive.<\/p>\n Faced with these frustrations, several technologies have been used over the years to increase business intelligence speed and flexibility. Each has provided valuable progress, but has some downside that prevented it being used on a more general basis. A tipping point has arrived, however, and a combination of these approaches holds out the promise of a radically simpler BI architecture.<\/p>\n The most important is \u201cin-memory processing\u201d. All computer processing has always happened in live memory, but up until now, there have been severe limitations on how much data could be stored, and so all data has first to be retrieved from disk storage before it can be processed.<\/p>\n Over time, memory processing capabilities has expanded exponentially, in line with Moore\u2019s Law, doubling every few years. But disk access speeds have been limited by real-world aerodynamics, and have increased only by 13x or so over the last fifty years. The result has been an ever-widening gulf between the speed of processing data and retrieving it from disk. Today, it can be up to a million times slower to get data from disk than from live memory.<\/p>\n But imagine now that the chef doesn\u2019t already have the ingredients ready. Given the slow relative speed of disk access, it\u2019s as if the closest supermarket was on the planet Mars, and the ingredients had to travel months by rocket before each and every meal.<\/p>\n Database vendors have taken every approach possible to increase disk access speeds, for example by predicting what data is most likely to be needed and caching it in advance (the equivalent of pre-stocking a larder in the restaurant, full of the ingredients that are most requested). But the whole point of a data warehouse is to be able to ask any question —\u00a0 the equivalent of being able to order any meal in the restaurant — and so you have to go back to the supermarket on Mars on a regular basis.<\/p>\n Up until recently, it\u2019s simply been too expensive to store data anywhere other than disk. But the price of memory has plummeted over the last two decades, and 64 bit addressing has radically increased how easy it is to access. Just ten years ago (when we first defined the current BI best practices) the price of one megabyte of live memory was around one dollar. Now it\u2019s over a hundred times less: below one cent, and still falling fast. This is equivalent to something shrinking from the size of the Statue of Liberty down to a Chihuahua: it would be strange indeed if this didn\u2019t have an impact on how we create our BI architectures.<\/p>\n If the whole data warehouse could be stored in-memory, we could make the whole system much faster and much simpler \u2013 and eliminate the need for disk-based storage altogether. We\u2019d no longer have need for database optimizations like aggregates and indexes \u2013 and eliminating these simplifies the data loading process, and allows us to store more data in that limited, valuable memory space.<\/p>\n But in-memory processing alone only gets you so far \u2013 to get the full value of in-memory processing, we want to pack in as much data as possible, and to do that, we can turn to a complementary technology: column data stores<\/a>.<\/p>\n Today\u2019s relational databases are row-based: each new set of data is written into the next-available memory space. This is fast, which is essential for high-volume transactional applications writing to slow disks. But there are downsides for storing analytic data in a row-based structure, in terms of storage efficiency and query speed.<\/p>\n Let\u2019s use an analogy to illustrate the difference between the systems: I employ a \u201crow-based\u201d filing system at home. I open each day\u2019s mail, take a quick look, and then put it on top of a big pile in the corner of my bedroom.\u00a0 At one level, it\u2019s an extremely efficient system: it\u2019s very fast to \u201cwrite\u201d to the database, and if I want to find all the papers I received on a particular date (a \u201ctransaction\u201d), I can find it pretty quickly.<\/p>\n But if I want to do some \u201canalysis\u201d, such as finding my last five bank statements, it\u2019s slow and painful: I have to systematically go through the whole pile (a \u201cfull table scan\u201d). I could make things faster by, say, adding yellow post-it notes to the corners of bank statements, so I can go straight to that type of document (a \u201cdatabase index\u201d), but that would create extra work and complicate the system.<\/p>\n My (far more organized) wife uses a \u201ccolumn-based\u201d filing system. When she receives her mail, she takes the time to sort out the documents and allocate them to separate folders. It\u2019s initially slower to store information, but it\u2019s much, much faster when she wants to find all her bank statements.<\/p>\n Column databases store data more efficiently, and allow greater compression, because you store similar types of information together. For example, I get paid the same amount each month, so rather than storing the same pay slip twelve times in the file, I could simply store it once, with a link to each month, and add an exception for my annual bonus. The result is that you can store between ten and a hundred times more data in the same physical space, shrinking the data warehouse back down to a size similar to the raw data used to create it (see diagram below). This in turn reduces the amount of memory you need to scan for each query and increases query processing speed.<\/p>\n Commercial column databases such as Sybase IQ have been around for a long time, and have proved their value, particularly in very high-volume data warehousing environments. But the extra up-front loading time, compared with the slow write-time to disks, has limited their use in the market.<\/p>\n Now let\u2019s imagine combining the two technologies, with an in-memory, column database. Because it\u2019s compact, we can now store the entire data warehouse in memory. And because it\u2019s fast, loading times are no longer a problem. We now have the best of both worlds without the downsides of each: the equivalent of being able to store the whole supermarket in the chef\u2019s kitchen.<\/p>\n But we haven\u2019t finished yet. We can bring three other data warehouse optimization techniques into the mix: analytic appliances, in-database algorithms, and incremental loading.<\/p>\n To go back to the filing analogy: it\u2019s the equivalent of my wife and I both working on our finances. If we had to work off the same pile of \u201crow-based\u201d documents, we\u2019d constantly be in each other\u2019s way. But with separate \u201ccolumn-based\u201d folders, my wife can look through the documents of one bank while I look through another, or we can split the folder in two and each look at one half of them (\u201cpartitioning\u201d).<\/p>\n We can also radically improve query speed by doing as much work as possible directly in the database. Today, if you want to do some statistical analysis, you typically have to do a query on the data warehouse, extract a large set of data to a separate statistics tool, create a predictive model, and then apply that to your data.<\/p>\n To take an overly simplistic example: imagine you wanted to rank one million customers by their cumulative revenue. if ranking is available directly in the database engine, you only have to extract one line of data \u2013 the result \u2013 rather than a million rows.<\/p>\n Of course, by having all the required data in-memory, and with the support of massively parallel processing, we can imagine far more sophisticated operations than just ranking. For example, accurate profitability and costing models can require huge amounts of processing. Being able to do it directly in the in-memory database would take a fraction of the time it typically does today.<\/p>\n Using a separate appliance also allows us to simplify our architecture: we can add a BI server in the same machine, without the need for a separate local calculation engine.<\/p>\n Next, instead of finding and updating data values in our data warehouse, we\u2019ll just incrementally replicate new data from the operational system as soon as it\u2019s available, and add a time-stamp. For each query, we simply ignore any out-of-date data. The loading process becomes much simpler and faster: we can just add new data as it arrives without having to look at what\u2019s already in the data warehouse. This also provides a big auditing and compliance advantage: we can easily recreate the state of the data warehouse at any point in time.<\/p>\n Since we\u2019re only loading new data, and we can do so in real-time, we no longer need an operational data store, and can eliminate it from our architecture.<\/p>\n It\u2019s worth noting at this point the virtuous circle created by these different technology advances working together. Different vendors in the industry typically concentrate on combining one or two approaches. Each provides an improvement, but combining them all is the real opportunity: together, they provide all the upside benefits while mitigating the downsides of each technology \u2013 and lead to the real tipping point, where you can realistically give up the disk-based storage.<\/p>\n Thanks to in-memory processing, column databases, hardware optimization, in-database calculations and incremental loading, we\u2019re left with the \u201ctransistor\u201d version of business analytics. It does the same thing as the previous architecture, but it\u2019s radically simpler, faster and with a more powerful architecture.<\/p>\n This diagram represents the \u201cnew best practice\u201d possibilities of the new generation of analytic platforms. But we can go even further, and start thinking about the \u201ctransistor\u201d phase of analytics \u2013 what new things might be possible in the future?<\/p>\n If we add ACID compliance<\/a> (a set of properties required for reliable processing of operational data), and the option of using row and\/or column storage in the same database, then we could use the same platform for both transactional and analytic processing. Backup storage would be provided by solid-state devices, so you could always recreate your in-memory environment.<\/p>\n This would have several advantages. First, we\u2019ve talked a lot about having a \u201csingle source of the truth\u201d in the last decades of business intelligence \u2013 if the transactions and the analytics are happening off the same set of data, then we can get a lot closer to our goal.<\/p>\n Second, we get closer to \u201cactionable intelligence\u201d, the ability to provide people people with analytic information as a seamless part of their operational activities, in time to make a difference (e.g. predict that the store will be out of stock tomorrow, and arrange a new delivery, rather than just telling us that the store ran out of stock yesterday).<\/p>\n Third, the new architecture is surprisingly well adapted to real-world business applications, which typically require much more than simple read-and-write operations. Application designers would no longer have to create complex logic above the database layer to do things like budget allocations or supply-chain optimization \u2013 these could use the superior analytic power of the analytic engine directly within the new platform.<\/p>\n We could then extend this architecture \u2013 putting it in the cloud would make mobile business intelligence, extranets, and data collaboration (both inside the organization and across the \u201cbusiness web\u201d) easier and simpler.<\/p>\n There are also enterprise information management advantages. For example, one common business frustration with BI has been how hard it is to compare corporate data stored in the data warehouse with personal or external data. Today, business people have to extract a large download from the data warehouse, and manually combine it with other data using Excel. This adds extra complexity and work, and leads to information governance issues.<\/p>\n With the new architecture, there\u2019s no longer any need for painful staged uploads to the data warehouse \u2013 we can create a \u201csandbox\u201d area in the analytic platform, let people upload their data in simple row format, and combine and analyze as they wish, using the same, standard, secure corporate infrastructure.<\/p>\n It also turns out that column databases do a good job of storing text data for easy search and retrieval, and other forms of data and algorithms (XML, hadoop) could potentially use the same infrastructure.<\/p>\n There\u2019s one key thing to note at this point: the diagram seems to imply that \u201cdata warehousing\u201d is no longer necessary. But nothing could be further from the truth. Reality is, and always will be, messy. The core need to transform, integrate, and model data, across a wide variety of different sources, is as important as ever.<\/p>\n Data integration, metadata, and data quality issues are business problems that don\u2019t magically disappear with a new technical infrastructure. But we can use the power of the in-memory calculations to do enterprise information management in real-time, rather than batch. It becomes more practical to integrate data on the fly (\u201cvirtual data warehousing), and we can take the data quality algorithms we use today (fuzzy matching, comparisons to master data, etc.), and execute them as transactions happen. We can store both the raw data, and the \u201cvalidated\u201d or \u201ccorrected\u201d values. This allows us to flexibly decide, at query time, how much reliability we need for a particular analytic need.<\/p>\n The last decade of BI best practice data warehouse architectures is rapidly becoming obsolete, as a combination of existing technologies comes together to provide a tipping point. Every vendor in the market includes some combination of in-memory, column databases, appliances, in-db calculations, map-reduce-type algorithms, and combining operations and analytics in their vision. The obvious conclusion is that the real vision is to use all of these technologies at the same time.<\/p>\n The new analytic platforms won\u2019t magically cure everything that plagues today\u2019s BI projects, but will lead to two big changes:<\/p>\n The last decade of traditional business intelligence \/ data warehousing best-practice infrastructures is rapidly becoming obsolete, as new technologies come together to provide a once-in-a-decade tipping point.<\/p>\n","protected":false},"author":2,"featured_media":2810,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[2,3],"tags":[160,198,204,213,269,271,351,560,595,717,911,1046,1072],"class_list":["post-12193","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-best-practice","category-bi-20","tag-bi","tag-business-analytics","tag-business-intelligence","tag-businessobjects","tag-column-database","tag-column-store","tag-data-warehousing","tag-hana","tag-in-memory","tag-map-reduce","tag-sap","tag-sybase-iq","tag-timo-elliott"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/newbiarchitecturecover.jpg?fit=668%2C300&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p3X9RF-3aF","_links":{"self":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts\/12193","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/comments?post=12193"}],"version-history":[{"count":0,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts\/12193\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/media\/2810"}],"wp:attachment":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/media?parent=12193"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/categories?post=12193"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/tags?post=12193"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image1.jpg?resize=690%2C364&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image2.jpg?resize=690%2C385&ssl=1\" "\\"image\\"")
<\/p>\n\n
![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image3.jpg?resize=690%2C420&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image4.jpg?resize=300%2C155&ssl=1\" "\\"image\\"")
This leads to tough architecture choices. One way of imagining the consequences is to compare it to a chef cooking a meal. If — like on the TV cooking shows — all the ingredients are already prepared and sitting on the counter-top, it\u2019s very quick and easy to create the meal. This is the equivalent of \u201cin-memory processing\u201d once all the required data is available.<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image5.jpg?resize=690%2C390&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image6.jpg?resize=690%2C410&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image7.jpg?resize=690%2C390&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image8.jpg?resize=690%2C284&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image10.jpg?resize=66%2C103&ssl=1\" "\\"image\\"")
By building a massively-parallel machine specifically for data warehousing, we can again radically increase the speed we can access and manipulate data. In particular, column databases are very well adapted to parallelization: because each column is stored in a separate area of memory, aggregates can be efficiently handed off to a separate processor, or even partitioned across several processors.<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image11.jpg?resize=690%2C226&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image12.jpg?resize=690%2C369&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image13.jpg?resize=690%2C303&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image14.jpg?resize=690%2C168&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image15.jpg?resize=690%2C252&ssl=1\" "\\"image\\"")
<\/p>\n![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2011\/03\/image16.jpg?resize=690%2C370&ssl=1\" "\\"image\\"")
<\/p>\nIn conclusion<\/h3>\n
\n
Other resources:<\/h3>\n
\n