![\"hana-and-hadoop\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2013\/08\/hana-and-hadoop.jpg?resize=690%2C310&ssl=1\" "\\"hana-and-hadoop\\"")
<\/p>\n<\/p>\n
Like most media organizations around the world, the Toronto-headquartered The Globe and Mail<\/a> has struggled to make a profitable transition<\/a> from physical newspapers to online journalism. But now a combination of Hadoop and SAP HANA in the cloud is helping make critical decisions about how and when to charge readers for online access to articles.<\/p>\n In print for 167 years, The Globe is Canada\u2019s largest newspaper, with over 300 journalists covering national, international, business, technology, arts, entertainment and lifestyle news for around 3.5 million readers a week across the country. Over the last decade, the company has invested in comprehensive data gathering and analysis systems, starting with SAP ERP in 2002 and a full enterprise data warehouse using SAP BW<\/a> in 2007.<\/p>\n In early 2012, data analysis became an urgent business priority because of the company\u2019s paywall project<\/a>. The company knew casual readers were coming to the web site and needed to work out how many articles the company should allow them to read before asking them to pay.<\/p>\n Sandy Yang, a functional analyst at the Globe and Mail, explained the problem: \u201cIf we set the bar too high, we won\u2019t have enough people to pay for our content, but if too low, they might never come back, and then we might lose a big chunk of our advertising revenue.\u201d The ideal solution be where \u201csome people shouldn\u2019t even know we have a pay wall, but some should think that their money is well spent. We need to find the right balance by analyzing user behavior.\u201d<\/p>\n The company uses Omniture to get insight into which articles readers are interested in and key statistics such as the number of page views per period or unique visits per period per section. But answering more complex \u2013 and important \u2013 questions required further analysis on the raw clickstream data. The internal IT teams first tried to import the web data from Omniture into a traditional relational database. But the data was complicated, stored in tab delimited text files with millions of lines, each having around 500 fields, and was growing at a rate of several gigabytes a day. The company turned to Hadoop to process the web data, but wasn\u2019t ready to buy and maintain its own servers, so used Amazon\u2019s Elastic MapReduce Architecture and stored the results in Amazon S3.<\/p>\n But that didn\u2019t solve all the analysts\u2019 problems. Yang explains: \u201cThe result is a whole lot of numbers. Every time a job finished, I had to add column headers and reformat the data to explain what it meant. And as soon as I handed it over to the business, they said \u2018OK, it looks good, but what if\u2026.?\u2019 I had to explain that it was a batch process, and that I couldn\u2019t drill down and give the answer immediately. I hated to have to answer like that, but I didn\u2019t have better options.\u201d<\/p>\n Figure 1: The Globe and Mail paywall project architecture, featuring Hadoop on Amazon AWS and SAP HANA<\/em><\/p>\n Then Yang discovered SAP HANA One, a version of the company\u2019s new in-memory platform that runs in the Amazon Cloud. \u201cIt was so simple we didn\u2019t think it was an SAP product! \u00a0HANA One bridged the gap between our inexplicable big data and our incredibly creative business people\u201d. The speed of the product met expectations: \u201cthe real-time aspect instead of batch processing was delivered as advertised.\u201d And HANA One came at the right price: \u201cThe total cost was just $3.50 an hour for visualization of data, instant response from user requests and more.\u201d<\/p>\n Setting up the system took less than four part-time days. Yang demonstrated it to the marketing teams who were instantly impressed with the big leap in data transparency and how easy it was to use SAP HANA Studio to visualize the data: \u201cwith its user-friendly client interface and fast processing, people saw numbers and charts within seconds, so big data was no longer formidable to them.\u201d<\/p>\n Figure 2: An example of a correlation analysis using SAP HANA Studio on the Globe and Mail clickstream data preprocessed in Hadoop<\/em><\/p>\n Yang found that starting up the server with the previous data only takes 15 minutes: \u201cI can use it whenever I want, and all I pay for is the time we use it, nothing more. For small businesses, and companies with no budgets, that\u2019s extremely important. In December, I spent less than $100 Canadian dollars \u2013 $25 for HANA One and $63 for AWS cluster servers \u2013 that\u2019s all!\u201d This helped make the implementation an easy business decision: \u201cUsually, you build a business case and then buy the products and then implement it. With HANA One, business can make its own case and we don\u2019t need to buy the product upfront \u2013 we pay as we go.\u201d<\/p>\n For more details about the Globe and Mail\u2019s Hadoop and SAP HANA One project on Amazon AWS, watch the on-demand web seminar<\/a> or read an extended interview with Sandy Yang<\/a> by the InsiderProfiles web site. And find out how easy it is to set up your own SAP HANA One solution in the cloud<\/a>.<\/p>\n![\"globeandmail5\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2013\/08\/globeandmail5_thumb.jpg?resize=632%2C384&ssl=1\" "\\"globeandmail5\\"")
<\/a><\/p>\n![\"globeandmail83\"](\"https:\/\/i0.wp.com\/timoelliott.com\/blog\/wp-content\/uploads\/2013\/08\/globeandmail83_thumb.jpg?resize=619%2C367&ssl=1\" "\\"globeandmail83\\"")
<\/a><\/p>\n