{"id":12462,"date":"2015-05-22T19:55:07","date_gmt":"2015-05-22T18:55:07","guid":{"rendered":"http:\/\/timoelliott.com\/blog\/?p=7318"},"modified":"2015-05-22T19:55:07","modified_gmt":"2015-05-22T18:55:07","slug":"top-big-data-challenges-revisited","status":"publish","type":"post","link":"https:\/\/timoelliott.com\/blog\/2015\/05\/top-big-data-challenges-revisited.html","title":{"rendered":"Top Big Data Challenges Revisited"},"content":{"rendered":"<p><a href=\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2015\/05\/orange-brain-path.jpg?ssl=1\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-7321\" src=\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2015\/05\/orange-brain-path.jpg?resize=608%2C385&#038;ssl=1\" alt=\"walk through brain deloitte sap colors\" width=\"608\" height=\"385\" \/><\/a><\/p>\n<p>We\u2019ve now been wrestling with Enterprise Big Data for a few years. Here\u2019s a summary of our progress so far.<\/p>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/phBtD4HBVEA\" width=\"608\" height=\"400\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<h3>There\u2019s still a lot of technology to learn<\/h3>\n<p>There\u2019s a lot of new technology to master, and it&#8217;s changing fast. There are new in-memory platforms like SAP HANA and open source projects like Hadoop and Spark. There are new techniques such as graph databases, text analytics, spatial processing, machine learning, and many more.<\/p>\n<p>The good news is we\u2019re <strong>now past the peak of the Hype Curve<\/strong>. In other words, people have had the chance to experiment and carry out pilots. They\u00a0now have a better idea of what the new technologies can and can\u2019t do.<\/p>\n<p>When people adopt new technology, they tend to do <strong>old things in new ways<\/strong>. But now that people have had a closer look, they&#8217;re finding\u00a0<strong>new things to do<\/strong>.<\/p>\n<p>For example, there\u2019s a lot less less talk about <strong>ripping-and-replacing<\/strong>\u00a0traditional analytics and more about <strong>wrapping-and-renewing: <\/strong>using Big Data to extend and innovate existing solutions.<\/p>\n<h3>The right people are still hard to find<\/h3>\n<p>Implementing Big Data means <strong>getting the right people with the right skills<\/strong>. It&#8217;s still hard to find people who really know these systems.<\/p>\n<p>The good news is that companies have been able to turn to existing analytics staff, who typically <strong>jump at the opportunity to learn new marketable skills<\/strong>. This is helped by the quantity of free training resources available online, such as <a href=\"https:\/\/open.sap.com\/\" target=\"_blank\">open.sap.com<\/a>\u00a0and the <a href=\"https:\/\/www.youtube.com\/user\/saphanaacademy\" target=\"_blank\">SAP HANA academy<\/a>. These courses explain the detailed technical steps to get value out of <a href=\"https:\/\/www.youtube.com\/watch?v=qKV_HzZigjI\" target=\"_blank\">both in-memory and Hadoop<\/a>.<\/p>\n<p>There\u2019s also a shortage of people who can get the most out of all this new data. Harvard business review called Data Scientist <strong>the sexiest job of the 21st century<\/strong> (it certainly sounds sexier than \u201cactuary,\u201d which is perhaps the closest 20th century equivalent!).\u00a0Data Scientists have deep analytic and statistical skills combined with knowledge of the business. They are in high demand and they command high salaries.<\/p>\n<p>What\u2019s new is that technology is helping to remove some of the the bottlenecks. For example there are now <strong>easier to use, <a href=\"http:\/\/scn.sap.com\/community\/predictive-analytics\/blog\/2014\/03\/11\/sap-infinite-insight--introduction-and-overview\" target=\"_blank\">more automated predictive analytics tools<\/a><\/strong> that can be deployed by, say, marketing staff looking to optimize campaigns. This frees up overloaded data scientists for the most strategic projects. And companies are starting to offer \u201c<strong>data science as a service<\/strong>.\u201d For example, SAP has <a href=\"http:\/\/www.sap.com\/solution\/big-data\/software\/services.html\" target=\"_blank\">teams of people<\/a> with advanced degrees in statistics who work on co-innovation projects with customers.<\/p>\n<h3>The right business case is key<\/h3>\n<p>One of the biggest challenges is still building the initial business case for Big Data projects in enterprise environments. And as usual there are two different approaches: either <strong>cutting\u00a0costs, or creating new opportunities.<\/strong><\/p>\n<p>One of my favorite cost-saving examples is a large european airline. They needed to be\u00a0able to store and query large amounts of detailed historical information for legal discovery reasons. Initially,\u00a0they would back it up to tape, and reload it to the data warehouse only when there was a court case. But that required a lot of work so they ended up leaving it in\u00a0the data warehouse even though it was seldom used. <strong>By moving that data to a Hadoop platform, they were able to save a lot of money,<\/strong> and the relatively slow query speed wasn\u2019t a concern for this use case.<\/p>\n<p>Another example is Molson Coors in Canada. They <a href=\"http:\/\/www.sap.com\/customer-testimonials\/consumer\/molsoncoors.html\" target=\"_blank\">moved their BW system to HANA<\/a>, not to speed up queries (they were already using the in-memory BW accelerator) but because they had increasing analytic demands, yet limited BI resources. Using SAP HANA vastly simplified the system, resulting in <strong>less data storage, lower maintenance, and reduced development effort.<\/strong><\/p>\n<p>The second approach is to go for something high-value that the company has always wanted to do, but it just wasn\u2019t feasible \u2014 and a lot of these have to do with improving\u00a0customer service.<\/p>\n<p>For example, Center Point Energy in Houston used SAP HANA <a href=\"http:\/\/1h9mo84f79012h5z12640ehs-wpengine.netdna-ssl.com\/wp-content\/uploads\/2015\/02\/SAP-HANA-Effect-Episode-6.pdf\" target=\"_blank\">to create<\/a> <strong>a \u201cmind-reading\u201d call center application<\/strong>. It identifies callers by their call ID, then runs a predictive algorithm using two years of customer history to choose among\u00a040 different reasons the customer could be calling &#8212; all in less than a second. It then directs them to the\u00a0most appropriate service, and provides all the information the operators need to handle the call. The result is that <strong>customers get better, faster service at a lower cost to the company.<\/strong><\/p>\n<p>The project wasn\u2019t new \u2014 it was something they had wanted to do for a long time, but previous attempts had failed. In order to be feasible, the results had to be available in less than five seconds. Using traditional systems it took over a minute and a half \u2014 far too slow to be useable on the call. And precalculating the information didn\u2019t work either \u2014 not only was all that processing and storage expensive, customers were often calling about a bill they had tried to pay just five minutes before.<\/p>\n<p>In all these\u00a0cases, the initial projects generated buy-in from the business users, and the\u00a0initial investments were\u00a0then leveraged for other applications at a lower marginal cost.<\/p>\n<h3>Integrating with existing systems is more important<\/h3>\n<p>Most enterprise Big Data project today are in standalone silos, with limited links to the existing corporate infrastructure. For example, while it&#8217;s clear that <strong>Hadoop is\u00a0enterprise ready<\/strong> in that it\u00a0can be used for valuable\u00a0projects, it&#8217;s also clear that it has\u00a0to be a lot\u00a0<a href=\"https:\/\/timoelliott.com\/blog\/2015\/04\/sap-embraces-hadoop-in-the-enterprise.html\" target=\"_blank\"><strong>enterprise-readier<\/strong><\/a>. In order to get the full value of these new technologies,<strong> it\u2019s vital to connect them to other corporate data<\/strong>. For example, it\u2019s pointless to track your Facebook likes without accounting for your marketing spend and whether they actually have any effect on sales.<\/p>\n<p>Truly integrated Big Data is still at the planning stage in most organizations. Recently, I\u2019ve been in a lot of meetings with customers who are debating their strategic information roadmaps. They\u2019re looking for <strong>new best-practice architecture blueprints<\/strong> that combine the new big data systems not only with traditional analytics, but also with their core business applications.<\/p>\n<h3>New business models are the next big opportunity<\/h3>\n<p>To get the full value of big data, you have to change your business processes, and change management is notoriously difficult. But of course, it\u2019s also the big opportunity. <strong>Because you can now measure new things in new ways, you can create new products and services<\/strong>.<\/p>\n<p>For example, Vodafone Netherlands\u00a0was able to <a href=\"https:\/\/timoelliott.com\/blog\/2015\/01\/vodafone-netherlands-new-products-made-simple.html\" target=\"_blank\">use\u00a0predictive analytics<\/a> to figure out the right people to target for <strong>discounted\u00a0roaming during the ski season<\/strong>. And <a href=\"http:\/\/scn.sap.com\/community\/hana-in-memory\/use-cases\/blog\/2014\/05\/20\/kaeser-kompressoren\" target=\"_blank\">Kaeser Compressor<\/a> used the data coming their equipment\u2019s sensors to <strong>create new online predictive maintenance services.<\/strong><\/p>\n<h3>Conclusion<\/h3>\n<p><strong>Big Data has come a long way. <\/strong>The fact that\u00a0we&#8217;re now in the<strong> <a href=\"http:\/\/www.gartner.com\/technology\/research\/methodologies\/hype-cycle.jsp\" target=\"_blank\">trough of disillusionment<\/a><\/strong>\u00a0 shows\u00a0we&#8217;re one big step closer to the slope of\u00a0enlightenment and the plateau of productivity.<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We\u2019ve now been wrestling with Enterprise Big Data for a few years. Here\u2019s a summary of where we are dealing with its challenges.<\/p>\n","protected":false},"author":2,"featured_media":12825,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[2],"tags":[27,100,173,236,835],"class_list":["post-12462","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-best-practice","tag-bigdata","tag-analytics","tag-big-data","tag-challenges","tag-predictive"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2015\/05\/orange-brain-path-608x385-1.jpg?fit=608%2C385&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p3X9RF-3f0","_links":{"self":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts\/12462","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/comments?post=12462"}],"version-history":[{"count":0,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts\/12462\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/media\/12825"}],"wp:attachment":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/media?parent=12462"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/categories?post=12462"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/tags?post=12462"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}