<\/p>\n
I was recently asked some questions by a journalist for Ziff Davis Media for an article on Big Data Trends. Here are my answers….<\/p>\n
What are the four biggest trends in big data use this year?<\/b><\/p>\n
1. The rise of Big Data Discovery<\/a>, that combines three of the hottest trends of the last few years in analytics: Big Data, Data Discovery, and Data Science.\u00a0The emerging Big Data Discovery tools will be simpler to use than existing data science products, more accessible to a wider ranger of users, and offer more powerful manipulation of a wider variety of data sources. The tools will allow the rise of a new type of user: a \u201ccitizen data scientist\u201d.<\/p>\n 2.\u00a0Self-service data preparation or \u201cdata wrangling”. Getting reliable, consistent data has always been the achilles heel of Big Data projects. New tools such as\u00a0SAP’s new Agile\u00a0Data Preparation product<\/a>\u00a0make it easier for individuals to grab, merge, cleanse, and collect different data sets. They extend and complement traditional ETL (extract, transform, load) tools and hand-coded data movement scripts.<\/p>\n 3.\u00a0Large enterprises are realizing that Big Data is here to stay, and now needs to be integrated with their existing corporate systems \u2014 there\u2019s no point in analyzing social data, say, unless you can connect it directly to your marketing and finance applications<\/a>. This means a lot more focus on governance, with projects such as Apache Atlas<\/a>, and new ways of organizing corporate analytics communities<\/a>.<\/p>\n 4.\u00a0Big Data Ecosystems. So far, most big data systems have been used to support the activities of an individual organization. But I\u2019m starting to see the rise of big data applications that cross multiple organizations to allow optimizations across an entire industry. A couple of examples: \u00a0The\u00a0Smart Port Logistics platform created by the Hamburg Port Authority<\/a>. It is designed to connect all of the participants of the port, including\u00a0the shipping companies, trucking companies, customs officials, and even the truck car parks and retail outlets. By collecting, analyzing, and feeding back information in real time, the Port Authority helps all the participants become more efficient. And the cooperation of\u00a0Volkswagen, Shell, and SAP on a\u00a0connected car<\/a>\u00a0ecosystem.<\/p>\n (Choosing just four is tough! The other huge area is\u00a0data privacy and “algorithm ethics”)<\/p>\n Who is winning in big data architecture? What vendors and architectures are becoming standards?<\/b><\/p>\n So far, most of the examples of big data implementations have been in silos, separate from the rest of the business. But to truly succeed with\u00a0big data organizations will need a \u201cmulti-polar\u201d strategy that combines new big data platforms like Hadoop and Spark with traditional data warehousing and new HTAP<\/a> (hybrid transactional analytic systems).<\/p>\n For now, everybody is \u201cwinning\u201d, because it\u2019s hard to find a vendor connected to analytics that isn\u2019t thriving. As big data becomes a normal part of business process,\u00a0vendors that have strong strategic ties with their customers will have an advantage.<\/p>\n How can data brokers bring context to big data?<\/b><\/p>\n According to research by Forrester,<\/a>\u00a0on average 45% of the data business people use resides outside of the enterprise BI environments. This clearly means that there is a big and growing opportunity for traditional data brokers — but they are starting to get competition from new sectors.<\/p>\n For example,\u00a0more organizations are realizing that the data in their systems is valuable, and they are starting to investigate becoming data brokers themselves. In addition, organizations like Ariba<\/a>, who run the largest business network in the world, already provide services such as procurement benchmarking<\/a>. Since the acquisition of the company by SAP, there are plans to expand these services into new areas.<\/p>\n How should businesses adjust to take advantage of the multipolar analytics trend?<\/b><\/p>\n Big Data architectures are going to remain complex for the foreseeable future, since both technology and business needs will continue to evolve rapidly. It\u2019s essential that organizations go beyond just having a \u201cdata architecture\u201d plan and create a full information community that includes the business units and data scientists.<\/p>\n The days of top-down, command-and-control information infrastructures are over. A community-based approach is essential to find the right pragmatic tradeoffs that will have the best long-term benefits for the company.<\/p>\n What are three ways that big data can improve IoT initiatives?<\/b><\/p>\n 1.Big Data is inextricably linked with IoT, as part of the overall datafication<\/a>\u00a0trend: exposing previously invisible business processes that can then be analyzed. Sensors generate massive amounts of information that may be in \u201cpoly structured\u201d formats.<\/p>\n 2. Simple sensors plus sophisticated predictive algorithms can transform existing business processes and create new opportunities. One of my favorites is the creation of \u201cmagic carpets<\/a>\u201d that allow the elderly to live longer in their own homes. They use simple sensors coupled with predictive analytics to discover\u00a0differences between “normal” and “abnormal” behavior and gait.<\/p>\n