{"id":12220,"date":"2011-09-27T14:42:18","date_gmt":"2011-09-27T13:42:18","guid":{"rendered":"http:\/\/timoelliott.com\/blog\/?p=3428"},"modified":"2011-09-27T14:42:18","modified_gmt":"2011-09-27T13:42:18","slug":"why-in-memory-analytics-is-like-digital-photography-an-industry-transformation","status":"publish","type":"post","link":"https:\/\/timoelliott.com\/blog\/2011\/09\/why-in-memory-analytics-is-like-digital-photography-an-industry-transformation.html","title":{"rendered":"Why In-Memory Analytics is Like Digital Photography: An Industry Transformation"},"content":{"rendered":"

\"film_to_digital[3]\"<\/p>\n

When was the last time you had a roll of film developed?<\/p>\n

If you\u2019re reading this, you\u2019re no technology luddite, so it was probably already many years ago. Film \u2013 a technology that had been slowly improved and perfected over 200 years \u00a0— was replaced in less than a decade by a much faster, cheaper, and more convenient technology approach.<\/p>\n

In an earlier post, I outlined why I think we\u2019re on the brink of a real revolution in business analytics infrastructures<\/a>. This post draws out some of the parallels with the shift from analog to digital image processing, and the effects it had on the industry as a whole.<\/p>\n

Film Cameras Compared to Today\u2019s Data Warehouses<\/h3>\n

\"canon-camera\"<\/p>\n

Data warehousing today has a lot in common with the first photos I took fifteen years ago with my then-brand-new Canon Elan II camera:<\/p>\n

High Costs<\/h4>\n

Buying film and getting pictures processed got very expensive very quickly. Only a tiny minority of professionals could take as many shots as they wanted \u2013 the rest of us had to be choosy about our shots.<\/p>\n

Today\u2019s data warehousing is complex and expensive, and organizations have to ruthlessly prioritize which projects will be undertaken. Meanwhile, other worthy projects have to wait, business users have to rely on gut feel.<\/p>\n

Upfront Planning Required<\/h4>\n

In order to get the best results, you had to know in advance what kind of thing you were going to be taking shots of. Want to take a landscape shot? You needed 100 ISO and daylight film. An action shot in a hockey rink? 1600 ISO and tungsten lighting. A colorful tropical scene? High-saturation Fujifilm Velvia. A moody night shot of Paris? High-grain Kodak black-and-white xyz\u2026.\u00a0 And the film only came in 24 or 36 pictures \u2013 you just changed your mind about what you wanted to take picture of? Tough.<\/p>\n

Today’s data warehouses require you to decide in advance what data you\u2019ll want to access later \u2013 and if you change your mind, you have to go back to the source data and reload it with different transformations.<\/p>\n

Slow Feedback Loop<\/h4>\n

What you saw through the eye-piece was very different from the resulting picture. By the time you got the film back and realized that the exposure was wrong, or the framing a little off, there wasn\u2019t that much you could do about it. The slow feedback loop meant that lots of good pictures were missed, and that you had to take more pictures than you needed. Want a great shot of a high-contrast night scene? You needed to take multiple long exposures (with the associated high costs) in order to have a hope of getting what you wanted.<\/p>\n

By the time business people have had a chance to access the data, reports, and dashboards in your business intelligence solutions, you\u2019ve already put in huge amounts of efforts \u2013 and the business has moved on, and now has different needs. To try to avoid this, you try to add all the data you might<\/em> need into the data warehouse, but this increases the project cost and complexity.<\/p>\n

Arcane Knowledge<\/h4>\n

With enough training, skilled photographers understand all the variables involved in getting the right picture, and were able to get more consistent, reliable results. But even they had to rely on specialized film processors to actually get the results.<\/p>\n

Because of all the factors mentioned above, data warehousing today requires lots of arcane knowledge to be successful, and the people that know how to do it are in high demand and short supply.<\/p>\n

Digital Cameras Compared to New Analytics Platforms Powered by In-Memory and Other Technologies*<\/h3>\n

\"sony-digital-camera\"<\/p>\n

I purchased one of the first consumer digital cameras, the Sony DSC-F1. It transformed the way I took pictures, just as in-memory computing is transforming analytics.<\/p>\n

Cheap, Easy, Iterative Learning and Experimentation<\/h4>\n

The screen shows you exactly the shot that you\u2019re going to take, and digital cameras don\u2019t require film or processing, so the marginal cost of another picture is effectively zero. You can try things, and if they don\u2019t work out, you can instantly adjust your approach and try again. Things like f-stops still make a difference to the end result, but you can learn how it works in an intuitive way, rather than having to wade through mathematics. Previously essential accessories like light meters become unnecessary \u2013 you can get the effect you need through trial-and-error in the same time it took you to work out the light readings.\u00a0 We are all better photographers now \u2013 because we can experiment, and throw away 90% of our pictures without worrying about it.<\/p>\n

Once you have the row-level data in-memory, you can easily change your analytic view on the fly. You want to use a different attribute, rolled-up in a different way? Rather than having to reload and retransform the data, you can simply make a change to the metadata, and the users get the results they want. You can quickly and iteratively prototype your business intelligence solutions, rather than having to try to rigorously plan everything in advance. Faster, more convenient analytics results in better<\/em> analytics.<\/p>\n

Faster Innovation<\/h4>\n

The combination of photography and computing has transformed what you can do with photos. Here are just some of the things that were almost unimaginable with analog film:<\/p>\n