Hadoop is a Tank. In-Memory is a Plane. Why Would You Need Either?!

Hadoop is a tank. Built by a workers’ cooperative, it’s complex and adaptable, and can handle any terrain.

In-memory is a plane. Pricier to purchase, but it’s sleek and efficient and goes hundreds of times faster than the average car.

You currently have a sedan car and can drive just about anywhere you want to go. You wonder why the heck people think you’d be interested in buying a tank or a plane. Here’s why:

(1) Efficiency

It turns out that the costs of tank and plane travel fell through the floor – to the point where they’re now cheaper and easier alternatives for many trips:

• Instead of taking the long road around the mountain (e.g. storing bulk information in a traditional DW), you can just jump in your tank and drive straight over the top (e.g. storing information in Hadoop).
• Instead of driving slowly to the next state (e.g. run a long ERP batch job), you can jump in the plane and get there in a fraction of the time at a fraction of the cost (e.g. run the same job in-memory).

It’s just technological progress. Don’t be the person in the horse and buggy wondering why everybody else has moved to new-fangled motor cars — hadoop and in-memory are quickly becoming a standard part of information architectures.

(2) Opportunity

With the new vehicles, there’s now a lot more terrain to explore:

• Tanks can take you to places the roads just don’t go. Yes, some of it is boring wasteland without much to look at, but you can now visit some spectacular spots that would otherwise be inaccessible (e.g. sentiment analysis on massive datasets).
• Planes allows you to visit entirely different continents, or go to another state as easily as you visit the local store (e.g. analyze data in previously-unthinkable ways, or intervene in business processes while they’re still happening).

Maybe you’re happy with your daily commute, and don’t feel the need for anything new. But don’t be surprised if the best spots are taken when you do finally get around to visiting.

The Future

One of the reasons you can drive around so easily today is because we spent decades building a lot of expensive roads. Our information landscapes are the same – a lot of work went into your enterprise data warehouse and data flows.

But as people look at expanding to new territories, they’re realizing that tanks and planes don’t need roads. The drivers of the next generation of “Big Data” information landscapes will be a lot freer to travel where and when they want to go, without IT first having to build roads for them (and policing and coordinating that travel will be harder).

And which of these types of transport will you need in the future? All of them! A tank with wings or a personal helicopter may be possible one day. But right now, you should keep your car, start investing in new vehicles, and draw up a plan for a good integrated transportation network.