{"id":12409,"date":"2014-03-28T17:19:41","date_gmt":"2014-03-28T16:19:41","guid":{"rendered":"http:\/\/timoelliott.com\/blog\/?p=6547"},"modified":"2014-03-28T17:19:41","modified_gmt":"2014-03-28T16:19:41","slug":"thoughts-on-the-future-of-hadoop-in-enterprise-environments","status":"publish","type":"post","link":"https:\/\/timoelliott.com\/blog\/2014\/03\/thoughts-on-the-future-of-hadoop-in-enterprise-environments.html","title":{"rendered":"Thoughts on The Future of Hadoop in Enterprise Environments"},"content":{"rendered":"

\"enterprise<\/a><\/p>\n

Between answering questions this week at\u00a0BI2014<\/a>\u00a0in Orlando\u00a0and preparing to ask some at\u00a0Hadoop Summit<\/a>\u00a0in Amsterdam next week, here are some thoughts on where we are with Hadoop for the enterprise.<\/p>\n

The elephant in the room<\/b>. It\u2019s now clear that Hadoop\u00a0will play a key part of your future enterprise architecture. But how will it integrate with existing infrastructures and other new technologies?<\/p>\n

Growing pains.<\/b>\u00a0It\u2019s teenagers vs parents. Teenagers are justifiably proud of disrupting the way their parents did things. But they have a tendency to underestimate the value of their parents’ experience. As they get older, they\u2019re forced to deal with more of the boring necessities of life \u2014 and realize their parents knew a thing or two after all.<\/p>\n

As Hadoop matures beyond pixel-only business models (social websites, online video games) and inexpensive\u00a0data storage\u00a0it is being forced to embrace \u2014 and learn a few lessons from \u2014 the enterprise systems that drive real-world business.<\/p>\n

Old dogs are learning new tricks.<\/b>\u00a0Even without Hadoop, enterprise systems are poised for big disruptive changes. New in-memory systems<\/a> eliminate the need for separate operational and analytic environments, combining transaction integrity with breakthrough performance. The data is stored once, and every transaction is available for analysis the instant it is written. The increased simplicity and agility of the system means it\u2019s both faster and \u2014 despite the higher costs of memory \u2014 cheaper to run than traditional architectures.<\/p>\n

Simplicity and agility.<\/b> Both Hadoop and in-memory deliver enterprise architects the agility that has been sorely missing in the past.\u00a0Hadoop doesn\u2019t require upfront definition of data schemas. In-memory systems offer fast analysis without complex caching and aggregate tables.<\/p>\n

Changes to support the business can be made in metadata, with less need to physically move data around. Updates can be made faster and more iterative. The future is uncertain\u2014above all, you need an architecture than can change fluidly with new business opportunities.<\/p>\n

Not just new tech in old ways.\u00a0<\/b>Information (not your transactional systems) is the new foundation of your future architecture. The data you are storing now you will still have in 20 years even through your applications will long since have changed.<\/p>\n

This means that it’s about more than using new technology in old ways, such as replacing part of enterprise data warehouses with more flexible Hadoop “data lakes\u201d or adding faster in-memory data marts. It has to be about supporting the\u00a0business needs of the future<\/a><\/p>\n

Enterprises are looking for an “innovation platform” with real-time analysis tightly connected to operations in order to power more personalized customer experiences and flexible business models.<\/p>\n

Today’s complex choices.<\/b>\u00a0Companies want to take advantage of the cost advantages of Hadoop systems, but they realize that Hadoop doesn\u2019t yet do everything they need (for example, Gartner surveys show a steady decline in the proportion of CIOs that believe that NoSQL will replace existing data warehousing rather than augmenting it —\u00a0now just 3%). And companies see the performance advantages of in-memory processing, but aren\u2019t sure how it can make a difference to their business.<\/p>\n

The new technologies confound easy classification and the boundaries continue to blur. The elephant is starting to wear a tie, with projects to introduce transaction integrity to Hadoop. Enterprise systems are providing support for things like in-database MapReduce<\/a> and text analysis<\/a>.<\/p>\n

There’s no easy one-size-fits-all answer today. Different \u201cinnovation applications\u201d require different tradeoffs, based on the types of data, the ratio of analysis to action, and the need for speed. Here are some examples of how Hadoop can fit in with enterprise architectures today<\/a>.<\/p>\n

It\u2019s not just about technology.<\/b>\u00a0Organizations shouldn\u2019t work on architecture without also thinking hard about how their business models may look in the future. And the success of Hadoop in the enterprise space depends as much on ecosystems of enterprise-savvy vendors and partners as it does on technology.<\/p>\n

The future seamless experience<\/b>. Companies are looking for the “new best practice” of how to put together an end-to-end, enterprise-strength information architecture.<\/p>\n

Vendors are racing to support that architecture vision with a combination of new and tried-and-true technologies. The goal is to hide complexity by automating as much as possible the handoffs between the different data systems to provide a coherent system rather than companies having to duct tape everything together themselves. This will require enterprise vendors to embrace Hadoop as if it were their own technology \u2014\u00a0which may require culture changes for some.<\/p>\n

Vendors are also working on packaged \u201cnext generation\u201d applications that combine operations, analytics, enterprise-ready mobile interfaces, links to third-party data, and integration with industry networks.<\/p>\n

Trust no one. <\/b>The technology continues to evolve at a rapid rate. There has been lots of enterprise experimentation with Hadoop but few organizations have yet declared Hadoop a first-tier element of their enterprise architecture. In-memory processing systems are also in their infancy, just now making the transition from fast analytics to all-around business platforms.<\/p>\n

Best practice is being discovered as we speak \u2014 and may change fast in the light of new technology changes. Don\u2019t trust anybody that claims that they can tell you the <\/i>right way to do things unless they\u2019ve first spent a lot of time understanding your business.<\/p>\n

Good luck, and join us in Amsterdam for more discussion! (SAP\/Hadoop<\/a> presentation by John Schitka at 15h10 on Wednesday April 2nd)<\/p>\n

Other Links<\/strong><\/p>\n