Agile Analytics

One of the biggest and frustrating barriers to successful analytics projects is lack of agility.

Analytics systems must be built for constant change. The whole point of analytics is “actionable insight.” The information accessed is supposed to result in changes to business process, and this in turn should trigger new opportunities and new analysis.  Business intelligence is like a bicycle — if it’s not moving, it’s falling over.

And riding a bicycle takes a certain amount of agility. Analytics systems have been complex and slow while business change has been accelerating. In order to provide the core figures required to run the business, IT organizations felt that they had to protect the data warehouse from new or unverified sources.

It isn’t that IT doesn’t know how to do analytics “right” — they just couldn’t act fast enough to satisfy their business customers. The result was regular crashes each time the business went around a tight corner and the analytics bicycle couldn’t turn in time.

The good news is that analytic technology has made huge leaps forward. The biggest benefit of new in-memory systems isn’t speed, it’s simplicity. The data is stored just once, for both transactions and analytics (Gartner calls this Hybrid Transaction/Analytical Processing or “HTAP”). And there’s no longer any need for “materialized views” and “aggregate tables” to ensure acceptable response times. This has eliminated most of the overhead and time required to prepare reliable data to answer business questions.

In addition, systems like Hadoop have enabled the storage and access of “schemaless” information that doesn’t require the rigid, predefined structures associated with today’s data warehouses.

Now that the technology has changed, analytic processes and organizations also have to adapt, and we’re starting to see this process in leading edge organizations. Analytics competency centers, aware that they no longer have a veto over departmental IT investments, are taking steps to ensure that “self-service BI” doesn’t turn into “self-centered BI.”

At the last SAP TechEd Conference, I had the opportunity to host a session talking about Agile Analytics with my guests Ronald Konijnenburg, SAP Mentor and Principal BI Consultant for CGI and Andy Steer, UK Group CTO for itelligence.

We had a great discussion covering topics such as:

• The changes we’ve seen in the industry
• How the rise of data discovery tools has changed the analytics landscape
• How IT organizations don’t trust business users to mash up data themselves
• How IT has to change the way they support the business
• Examples of organizations that are making those changes
• Why BI is like a buffet (and you can bring your own veggie burgers)
• Data governance in the hands of business users, including data quality incentives
• The adoption of new agile BI methodologies
• Collaboration tools for analytics
• The blurring of BI and business roles
• The need to focus on business outcomes
• Data quality remains a big barrier
• Some businesses are not very agile
• Reporting is about “telling people what they already know” eg. compliance, statutory reporting.
• What are are the remaining big barriers to analytics? adoption, best practice, bringing together analytics and operations.

What do you think about Agile Analytics? What did we miss in our discussion? Comment below!