Are these CIOs deluded? No, the fact that the numbers continue to drop are a good indication that they understand the technology — and its limitations.<\/p>\n Note this DOESN\u2019T mean that enterprises don’t see value of Hadoop, or aren’t going to massively increase their investment in it. Quite the opposite. At this week\u2019s Hadoop Summit<\/a> in Amsterdam, I heard from large enterprises such as Deutsche Telekom, Centrica, EDF, HSBC, and ING bank that are making big strategic bets on Hadoop as a core data framework for the future<\/strong>.<\/p>\n But like the survey respondents, they believe that Hadoop ultimately \u201cadds more than subtracts.\u201d For example, Alasdair Anderson<\/a> of HSBC gave great, concrete example of the cost and flexibility advantages of Hadoop in his presentation \u201cHadoop-economics, the Invisible Hand of Big Data.\u201d But he also included a \u201cHealth Warning\u201d to use Hadoop where it makes sense:<\/p>\n \u201cThere\u2019s no relationship between the EDW and Hadoop right now \u2014 they are going to be complimentary. It\u2019s NOT about rip and replace: we\u2019re not going to get rid of RDBMS or MPP, but instead use the right tool for right job \u2014 and that will very much be driven by price.\u201d<\/p><\/blockquote>\n Ralph Kimball<\/a>, one of the key data warehousing pioneers, echoed these sentiments in a recent CloudEra webinar<\/a>. In a must-see presentation designed to explain Hadoop to data warehouse experts, he positively gushed over the new opportunities, but here\u2019s what he had to say in the Q&A section:<\/p>\n \u201cHere\u2019s a question that made me laugh a little bit, but it\u2019s a serious question: \u2018Well does this mean that relational databases are going to die?\u2019. I think that there was a sense, three or four years ago, that maybe this was all a giant zero sum game between Hadoop and relational databases, and that has simply gone away. Everyone has now realized that there\u2019s a huge legacy value in relational databases for the purposes they are used for. Not only transaction processing, but for all the very focused, index-oriented queries on that kind of data, and that will continue in a very robust way forever. Hadoop, therefore, will present this alternative kind of environment for different types of analysis for different kinds of data, and the two of them will coexist. And they will call each other. There may be points at which the business user isn\u2019t actually quite sure which one of them they are touching at any point of time.\u201d<\/p><\/blockquote>\n So Hadoop isn\u2019t going to \u201creplace\u201d EDW. But how about that notion that EDWs are legacy, to be maintained but not used for new analytic applications? Will EDWs become \u201cthe mainframes of 21st century\u201d as one Hadoop Summit attendee put it?<\/p>\n No. Ignoring the many advantages of Hadoop would be dumb. But it would be just as dumb to ignore the other revolutionary technology breakthroughs in the DW space. In particular, new in-memory processing opportunities have created a brand-new category that Gartner calls \u201chybrid transactional\/analytic platforms<\/a>\u201d (HTAP):<\/p>\n \u201cHybrid transaction\/analytical processing will empower application leaders to innovate via greater situation awareness and improved business agility. This will entail an upheaval in the established architectures, technologies and skills driven by use of in-memory computing technologies as enablers.\u201d<\/p><\/blockquote>\n HTAP isn\u2019t just a way of speeding up existing applications and analytics. The simplicity of the architecture (data is stored just once), and the flexibility of the platform means it has a lower total cost of ownership that traditional disk-based platforms<\/a>. This means it\u2019s also rapidly becoming the default platform<\/a> for new business application deployments, and the heart of new “real-time” applications.<\/p>\n This means there is no a three-way tug for new analytic uses — directly on the operational system with HTAP, Hadoop, or traditional DW (and\/or flexible data mashing with data discovery tools)<\/strong><\/p>\n No. This was a gratingly prevalent view at at Hadoop Summit, but it\u2019s firmly in the \u201cwishful thinking\u201d camp.<\/p>\n Yes, there are projects to make in-memory and ACID compliance part of the Hadoop Framework.\u00a0Storm and Flume mean you can start using Hadoop with streaming data.\u00a0YARN may turn into a \u201cgeneric app provisioning system via Linux containers.\u201d<\/p>\n Does this mean that you\u2019ll be able to do more with Hadoop in the future? Yes. Is it going to be easier to make applications? Yes. Is forty years of business process and data warehousing technology and expertise going be obsolete any time soon? No!<\/p>\n One small example, from Alan Gates<\/a>, one of the co-founders of Hortonworks, presenting the new ACID features in the next version of Hive, answering a question about whether this would enable OLTP support:<\/p>\n \u201cWe\u2019re not aiming at that, it\u2019s not what HIVE is good at, we don\u2019t think it makes any sense, and it would be a total fail if you did that.\u201d<\/p><\/blockquote>\n There are other projects for adding transactions to Hadoop, but integration with the corporate world will require support for standards like SQL, and the irony is that it will be a big challenge for Hadoop to support even 22yr-old SQL-92<\/a>. As analyst Merv Adrian<\/a> put it in a recent Gartner conference presentation: \u201cWhat is remarkable is that Hadoop does SQL. Just don\u2019t expect it to do it well\u201d. (More in Gartner\u2019s report: \u201cchoosing your SQL access strategy for Hadoop<\/a>\u201d)<\/p>\n The right answer for the future is the boring, pragmatic one that big data organizations like Hortonworks emphasize in their “modern data architecture<\/a>\u201d diagram. To make the most of your data, you will need all these proven and new technologies working seamlessly together. <\/strong><\/em><\/p>\n![\"gartner](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2014\/04\/gartnerdwhadoop_thumb.jpg?resize=608%2C382&ssl=1\" "\\"gartner")
<\/a><\/p>\nEmbrace Technology Change Outside of Hadoop, Too<\/h3>\n
![\"image\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2014\/04\/image_thumb.png?resize=610%2C470&ssl=1\" "\\"image\\"")
<\/p>\nBut isn’t Hadoop going to do all that?<\/h3>\n
The Architecture of the Future<\/h3>\n
![\"hadoop](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2014\/04\/hadoopmodernarchitecture_thumb.png?resize=608%2C495&ssl=1\" "\\"hadoop")
<\/a><\/p>\n