{"id":12300,"date":"2013-02-24T00:37:56","date_gmt":"2013-02-23T23:37:56","guid":{"rendered":"http:\/\/timoelliott.com\/blog\/?p=4693"},"modified":"2013-02-24T00:37:56","modified_gmt":"2013-02-23T23:37:56","slug":"gartnerbi-and-big-data-fear-loathing-and-business-breakthroughs","status":"publish","type":"post","link":"https:\/\/timoelliott.com\/blog\/2013\/02\/gartnerbi-and-big-data-fear-loathing-and-business-breakthroughs.html","title":{"rendered":"#GartnerBI and Big Data: Fear, Loathing, and Business Breakthroughs"},"content":{"rendered":"<p><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: inline; padding-top: 0px; border-width: 0px;\" title=\"big data\" alt=\"big data\" src=\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2013\/02\/big-data.jpg?resize=690%2C419&#038;ssl=1\" width=\"690\" height=\"419\" border=\"0\" \/><\/p>\n<p>At the Gartner Business Intelligence and Analytics Summit in Barcelona, there was near universal agreement about three things to do with \u201cbig data\u201d<\/p>\n<ol>\n<li>It\u2019s an awful term (but we\u2019re stuck with it)<\/li>\n<li>Whatever it means, it\u2019s a big deal, and requires big changes to traditional information infrastructures<\/li>\n<li>It will result in big new business opportunities<\/li>\n<\/ol>\n<h3>\u201cBig Data\u201d is a terrible term<\/h3>\n<p>Gartner analyst\u00a0<a href=\"http:\/\/twitter.com\/doug_laney\" target=\"_blank\">Doug Laney<\/a>\u00a0first coined the term \u201cbig data\u201d over <a href=\"http:\/\/blogs.gartner.com\/doug-laney\/deja-vvvue-others-claiming-gartners-volume-velocity-variety-construct-for-big-data\/\" target=\"_blank\">over 12 years ago<\/a> (at least in its current form \u2013 people have been complaining about \u201cinformation overload\u201d since Roman times). But the term\u2019s meaning is still far from clear and it was nominated the #1 \u201c<a href=\"http:\/\/www.languagemonitor.com\/high-tech-buzzwords\/top-tech-buzzwords-everyone-uses-but-dont-quite-understand-2012\/\" target=\"_blank\">tech buzzword<\/a> that everyone uses but don\u2019t quite understand\u201d (followed closely by \u201ccloud\u201d).<\/p>\n<p>When using the term, Gartner usually keeps the quote marks in place (i.e. it\u2019s \u201cbig data\u201d, not big data). Here\u2019s the definition provided by analyst <a href=\"http\/\/twitter.com\/ted_friedman\" target=\"_blank\">Ted Friedman<\/a> to \u201cde-hype\u201d the term during the summit keynote:<\/p>\n<blockquote><p>&#8220;Big Data&#8221; are high volume, velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision-making\u201d<\/p><\/blockquote>\n<p>Analyst <a href=\"https:\/\/twitter.com\/brazingo\" target=\"_blank\">Donald Feinberg<\/a> warned people that \u201ctalking only about big data can lead to self-delusion\u201d and urged people not to \u201csurrender to the hype-ocracy.\u201d He left left no doubt over where he stood on the use of the term: \u201cBig data doesn\u2019t mean MapReduce or Hadoop. Big data doesn\u2019t exist, it\u2019s meaningless, it\u2019s ridiculous\u2026\u201d The audience started applauding, to which he replied: \u201cWhy are you clapping?! Why do you all fall for it? Why do the vendors do it?!<\/p>\n<p>As SAP\u2019s <a href=\"https:\/\/twitter.com\/rosejason\">Jason Rose?<\/a> put it \u201chow can we demystify this? &#8230;easy, drop the \u2018big\u2019. Data has always been the key challenge in BI\u201d.<\/p>\n<p>For what it\u2019s worth, here\u2019s my tongue-in-cheek <a href=\"https:\/\/timoelliott.com\/blog\/2012\/12\/what-is-big-data.html\" target=\"_blank\">definition<\/a>, and it\u2019s worth noting that big data is quickly becoming the <a href=\"https:\/\/timoelliott.com\/blog\/2012\/12\/is-big-data-the-new-term-for-business-intelligence.html\" target=\"_blank\">default term<\/a> for what we used to call analytics or business intelligence.<\/p>\n<h3>But Big Data is a Big Deal<\/h3>\n<p>Despite the problems, Doug Laney noted that big data the most-searched-for term on Gartner.com.\u00a0 Why is it so popular? Maybe because it\u2019s so nebulous that people want to check if they have understood it. Or maybe because there\u2019s no other more precise term to indicate the new analytic opportunities. And maybe because \u201chype has a value\u201d as <a href=\"http\/\/twitter.com\/ted_friedman\" target=\"_blank\">Ted Friedman<\/a> put it: big data has proved to be a new opportunity to talk to business people about the power of analytics, and because everybody\u2019s searching for it, vendors would be crazy not to include it in their marketing.<\/p>\n<p>Conference attendees generally believed that the biggest opportunity for big data analysis was new insights from \u201cdark data\u201d that lies unused within organizations today. Gartner highlighted the dangers of implementing shiny new big data technology, separate from existing analytics infrastructures: \u201cDo not make your big data implementations siloed. Make them part of the overall strategy for BI.\u201d <a href=\"http:\/\/www.computerweekly.com\/news\/2240177374\/Dont-make-big-data-stand-alone-warns-Gartner\" target=\"_blank\">said<\/a> Ted Friedman. \u201cLink to stuff you are already doing. Don\u2019t make big data a standalone thing. And don\u2019t feel like you\u2019ve got to go out and buy a whole new technology stack.\u201d<\/p>\n<p>Analyst <a href=\"https:\/\/twitter.com\/rsallam\" target=\"_blank\">Rita Sallam<\/a>, in a session on data variety, gave some examples of the new opportunities:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" alt=\"\" src=\"https:\/\/pbs.twimg.com\/media\/BCbVqO7CAAExmhc.jpg:large\" width=\"480\" height=\"360\" \/><\/p>\n<p>Some of Gartner\u2019s public <a href=\"http:\/\/www.gartner.com\/newsroom\/id\/2313915\" target=\"_blank\">predictions<\/a> related to big data:<\/p>\n<ul>\n<li>By 2015, 65 percent of packaged analytic applications with advanced analytics will come embedded with Hadoop.<\/li>\n<li>By 2016, 70 percent of leading BI vendors will have incorporated natural-language and spoken-word capabilities.<\/li>\n<li>By 2015, more than 30 percent of analytics projects will deliver insights based on structured and unstructured data.<\/li>\n<\/ul>\n<h3>But What Does It Mean For The Business?<\/h3>\n<p>Donald Fienberg: \u201cRealize that big data is not about doing \u2018more\u2019 of the same thing \u2013 it\u2019s about doing things differently\u201d. \u201cThe major opportunities for big data are around ways to transform the business and disrupt the industry\u201d said Doug Laney. These included radically changing existing business processes, introducing new, more-personalized products and services, and \u201canswering chewy questions that weren\u2019t possible before.\u201d Some examples:<\/p>\n<ul>\n<li>NetFlix did deep analysis of their viewers\u2019 preferences, and used that to craft the new &#8220;House of Cards&#8221; TV series \u2013 a $100M investment<\/li>\n<li>New financial lenders are using big data to find untapped banking opportunities \u2013 including lending scores based on what you say on social media<\/li>\n<li><a href=\"http:\/\/www.passur.com\/\" target=\"_blank\">Passur<\/a> uses big data to provides real-time monitoring of air traffic, to potentially save millions of dollars per year. Today, pilots\u2019 estimated times of arrival are off by more than ten minutes ten percent of the time, and five minutes 30% of the time \u2013 knowing <em>exactly<\/em> when the planes will arrive means more automation, better operating efficiencies, improved security, etc.<\/li>\n<li><a href=\"http:\/\/www.enologix.com\/\" target=\"_blank\">Enologix<\/a> analyzes the chemical composition of new wines to predict wine spectator score, and offer advice on how to improve the score<\/li>\n<li>Dollar General, Kroger and other retailers provide data to partners to analyze, for &#8220;free strategic advice&#8221;<\/li>\n<li>Insurance companies are using text mining on previously-unexamined \u201cdark data\u201d on claims forms to sniff out indicators of fraud<\/li>\n<\/ul>\n<p><a href=\"http:\/\/se.linkedin.com\/pub\/mats-olov-eriksson\/b\/3b9\/71a\" target=\"_blank\">Mats-Olov Eriksson<\/a> of <a href=\"http:\/\/www.king.com\/\" target=\"_blank\">King.com<\/a>, a Sweden-based casual gaming site (card games, social games, etc), gave an entertaining presentation called \u201cBeyond Big and Data\u201d, hosted by recently-returned Gartner analyst <a href=\"https:\/\/twitter.com\/FrankBuytendijk\" target=\"_blank\">Frank Buytendijk<\/a>. It was illustrated with more cute kittens, puppies, and otters than I\u2019ve ever seen before in a technology conference!<\/p>\n<p><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: inline; padding-top: 0px; border-width: 0px;\" title=\"kingcom\" alt=\"kingcom\" src=\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2013\/02\/kingcom.jpg?resize=690%2C246&#038;ssl=1\" width=\"690\" height=\"246\" border=\"0\" \/><\/p>\n<p>Mat\u2019s title is \u201cSpiritual Leader, Data Warehouse\u201d, and he explained how the company collects massive amounts of log data about user activities, then uses a combination of Hadoop, traditional data warehousing, and BI cubes to optimize the player experience \u2013 and the company\u2019s profits.<\/p>\n<p>The company works with Facebook on games like <a href=\"http:\/\/apps.facebook.com\/candycrush\/\" target=\"_blank\">Candy Crush Saga<\/a>. The company\u2019s main revenue comes from in-app purchases: for example, you might get five free lives in a game, and if you run out you can either wait for a period, or pay a small sum to play immediately. To try to figure out what options generate the most revenue, and keep players in the game, the company currently processes 60,000 log files per day. The files contain around three billion \u201cevents\u201d per day, and the company expects that to rise to six billion by the end of this year. Mats explained that big data is best compared to a software development project, requiring hard-to-find skilled personnel. He characterized hadoop as a \u201cpowerful but immature five year old\u201d and explained that they used a MySQL data mart for low-latency access: \u201cHIVE is anything but low latency \u2013 it takes fifteen seconds before you even get an error message!&#8221;.<\/p>\n<p>One of the recent challenges the company has faced is information governance \u2013 a term they hadn\u2019t even used until six months ago. Mats noted that the European privacy laws were by far the \u201charshest\u201d they deal with.<\/p>\n<p>In a separate presentation, that I unfortunately didn\u2019t attend, <a href=\"https:\/\/twitter.com\/tfastner\" target=\"_blank\">Tom Fastner<\/a> of eBay gave an overview of his company\u2019s analytic systems, explaining that they process more than 100 Petabytes of data I\/Os every day, and a \u201csingularity\u201d keyvalue store on Teradata manages 36 Petabytes of data. The largest table is 3 Petabytes \/ 3 trillion rows, and only takes 32 seconds to scan. Here\u2019s a presentation from 2011 that talks about how eBay combines <a href=\"http:\/\/www-conf.slac.stanford.edu\/xldb2011\/talks\/xldb2011_tue_1055_TomFastner.pdf\" target=\"_blank\">structured, semi-structured, and unstructured data<\/a>.<\/p>\n<p>The inimitable <a href=\"https:\/\/twitter.com\/nstevenlucas\" target=\"_blank\">Steve Lucas<\/a> says of big data: \u201cwhen you see it, you know it\u201d. MKI, a bioscience company is <a href=\"http:\/\/www.sapvirtualevents.com\/sapphirenow\/sessiondetails.aspx?sId=2388\" target=\"_blank\">using big data to analyze genome data<\/a>, using a combination of hadoop, open-source \u201c<a href=\"http:\/\/www.r-project.org\/\" target=\"_blank\">R<\/a>\u201d algorithms, and SAP\u2019s in-memory platform, <a href=\"http:\/\/www.saphana.com\/\" target=\"_blank\">SAP HANA<\/a> (registration required).<\/p>\n<p><a href=\"http:\/\/www.sapvirtualevents.com\/sapphirenow\/sessiondetails.aspx?sId=2388\" target=\"_blank\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: inline; padding-top: 0px; border: 0px;\" title=\"image\" alt=\"image\" src=\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2013\/02\/image.png?resize=690%2C388&#038;ssl=1\" width=\"690\" height=\"388\" border=\"0\" \/><\/a><\/p>\n<p><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: inline; padding-top: 0px; border: 0px;\" title=\"image\" alt=\"image\" src=\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2013\/02\/image1.png?resize=690%2C388&#038;ssl=1\" width=\"690\" height=\"388\" border=\"0\" \/><\/p>\n<p><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: inline; padding-top: 0px; border: 0px;\" title=\"image\" alt=\"image\" src=\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2013\/02\/image2.png?resize=690%2C388&#038;ssl=1\" width=\"690\" height=\"388\" border=\"0\" \/><\/p>\n<p>Here\u2019s a summary of the MKI project:<\/p>\n<p><iframe loading=\"lazy\" src=\"http:\/\/www.youtube.com\/embed\/Y-zOq3nKjaA\" height=\"388\" width=\"690\" allowfullscreen=\"allowfullscreen\" frameborder=\"0\"><\/iframe><\/p>\n<h3>In Conclusion<\/h3>\n<p>Big data is a lousy term, but offers big opportunities in return for some big information infrastructure changes. What does the future hold? Let\u2019s hope less of the hype, and more of the business change\u2026<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>At the Gartner Business Intelligence and Analytics EMEA Summit in Barcelona, everybody agreed that: big data is a terrible term, but it&#8217;s a big deal, and results in big changes to business opportunities<\/p>\n","protected":false},"author":2,"featured_media":4686,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[5],"tags":[27,100,160,173,204,213,428,524,556,560,835,916,931,1035],"class_list":["post-12300","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-conferences","tag-bigdata","tag-analytics","tag-bi","tag-big-data","tag-business-intelligence","tag-businessobjects","tag-emea","tag-gartnerbi","tag-hadoop","tag-hana","tag-predictive","tag-sap-hana","tag-saphana","tag-summit"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2013\/02\/big-data-banner-1.jpg?fit=690%2C310&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p3X9RF-3co","_links":{"self":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts\/12300","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/comments?post=12300"}],"version-history":[{"count":0,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/posts\/12300\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/media\/4686"}],"wp:attachment":[{"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/media?parent=12300"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/categories?post=12300"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/timoelliott.com\/blog\/wp-json\/wp\/v2\/tags?post=12300"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}