Google has announced Google Refine 2.0, “a power tool for data wranglers”. It’s an open-source tool for cleaning and enhancing “messy” data sets, including cleaning up inconsistencies, transforming them from one format into another, and extending them with new data from external web services or other databases.

How does it shape up from a corporate point of view? Is this another step for Google into enterprise information management?

The new version includes:

“new extensions architecture, a reconciliation framework for linking records to other databases (like Freebase), and a ton of new transformation commands and expressions.”

The tool runs on your desktop (even though you’re accessing through a browser), so you don’t have to worry about data security.

Here are some videos about the new product. First, an introduction to how you can use the tool to do basic manual cleansing of data:

• Converting various badly-entered variants of “FFP” into “Firm Fixed Price”
• Helping “cluster” groups of similar data together using heuristics
• Using expressions to change distributions using log functions
• Identifying problems (zero values, errors between millions/billions, etc.)

Next, data transformations. The tool makes it easy to convert information in a basic HTML list into a nicely formatted table, using filtering and an expression language:

• Isolating certain rows
• Simple conversions, e.g. removing bolding
• Extracting part of the values to a new column
• Splitting existing values into new columns

The extractions used then be exported into a standard like JSON, and used to convert similarly-formatted datasets.

Finally, you can use Google Refine to augment your data with data from web services and how to link your data with databases such as Freebase:

• Calling web services to add geo-coding to address information
• Using Google’s language detection service to identify the language of different values
• Doing database joins with external data sources (Google calls this “reconciliation”)
• Freebase has a nice service that will automatically figure out what type of values you have in your data (e.g. movie names), and matches them to appropriate records, and offers you options in case of ambiguity (e.g. there are several movies containing the word “Terminator”)
• Once you have a match, you can choose from other fields (such as the movie release date, etc.)

Overall, it looks like Google Refine 2.0 is great free option for correcting and correlating small, real-world messy data sets from the web — but you have to be a JSON ninja to get the most out of it.

It’s clear that you could already use this in a corporate context — there are many one-off projects that require manual collection of information from different sources. And organizations suffer from just the same types of information fragmentation as the general web.

With a more user-friendly interface and (presumably) more robust scalability, this type of tool could be of great benefit for people trying to pragmatically cobble together  information from various data sources (a few spreadsheets, an external web site, the corporate data warehouse, etc.)

It’s not clear that corporate data is or ever will be a priority for Google — the entire enterprise information management market is a rounding error compared to their advertising empire. In the meantime, anything that can help people get the data they need, cheaply, should be welcomed.

This type of solution will never replace the need for a robust enterprise information platform, but the need for “messy” solutions to answer real-world business questions is a frequently-underestimated need in business analytics deployments, and Google Refine 2.0 looks like a great tool to add to the workbench.

Next week sees the start of the Nordic SAP BusinessObjects BI Forums in Stockholm (April 13), Copenhagen (April 15), Oslo (April 27), and Helsinki (April 29). Join us for these free events to find out:

“…how successful industry leaders turns data into insight-driven business decisions. Don’t miss out on this exclusive opportunity to learn from managers and experts how to optimize your business performance. This is the conference day of the year for business users with a professional interest in business intelligence (BI); information management (IM); enterprise performance management (EPM); and governance, risk, and compliance (GRC).”

I’ll be presenting the morning keynote in each location – view the video below to get a quick overview of the topics I’ll be covering.

You can register for the events here, and I look forward to seeing you there!

[Update:
Here's the keynote presentation as a pdf file, and as a ppt file
Here's the BI session presentation as a pdf file, and as a ppt file]

Looking for some free training in these difficult economic times?

Here’s a long, long list (OK, so there are “only” 96 of them) of free web seminars courtesy of the SDN BusinessObjects Web Seminar Series and Jason Cao. They cover almost every aspect of the BusinessObjects Intelligence Platform.

And if you don’t find what you’re looking for, you can register to be notified by email of upcoming BusinessObjects webinars and suggest topics for future webinars.

Recent and Upcoming Web Seminars

Archived web seminars:

• Dashboards and Data Visualization
• Information Management
• Innovations
• Intelligence Platform
• Query, Reporting, Analysis
• Search and Navigation
• Software Development Kits (SDKs)

Dashboards and Data Visualization

Information Management

Innovations

Intelligence Platform

Query, Reporting, Analysis

Search and Navigation

Software Development Kits (SDKs)

According to Albert Pang of IDC, the current economic conditions are forcing organizations to not only better manage corporate performance but also increase business transparency: companies are under increasing pressure to conduct operations effectively, profitably, and ethically — and prove it. To achieve this, IDC advises companies to integrate EPM, GRC, and business analytics.

The IDC paper, Helping Enterprises Succeed: Responsible Corporate Strategy and Intelligent Business Insights covers the following:

This Executive Insights paper describes how the combination of enterprise performance management (EPM) and governance, risk, and compliance (GRC) solutions is now more critical than ever for business success.

This document also discusses how EPM and GRC must be linked to business intelligence tools and supported by comprehensive services to ensure that the data generated is used appropriately. Services such as strategy management enable companies to more readily develop, measure, and adjust corporate strategy according to new compliance requirements and regulations. Then, as new regulations come up, companies can easily adjust and monitor their strategy via dashboards with clear reporting. Also presented are suggestions for working with IT service providers to implement strategic analytics-based EPM and GRC solutions.

You can access the white paper here (registration required)

1. Tame information chaos: Collect, manage, and integrate information, and fix any data quality problems (data quality, data integration, metadata management, etc)
2. Turn information into insight: Analyze the information, gain understanding of what is happening, and communicate it to others (query, analysis, reporting, etc.)
3. Use insight to drive change: Support processes for turning understanding into action (scorecards, budgeting and planning, profitability analysis, financial consolidation, analytics, etc.)

Category chaos

There’s general agreement that all these things are needed, but the exact definition and name of each category is hotly debated.

Is the whole thing called Performance Management or Business Intelligence? Which includes which? Or are they separate, but overlapping terms? And of course neither is possible without a solid information platform – is data integration part of BI? Is metadata management part of PM?

Convergence

Whatever the differences, the category edges are blurring, and there are fewer vendors that have products only in one area.

• Between PM and BI. Performance management and business intelligence are essentially two sides of the same coin: there’s no point of defining key performance indicators unless you can track them, and there’s no point in measuring anything unless you have an idea of what it should be.
• Between BI and IM. The prettiest chart in the world is useless unless the data in it can be trusted, and BI and search are starting to overlap.
• Between IM and PM. Compliance rules mean organizations want to be able to drill back to find where each financial figure came from and when it was last updated.

Time for a new name?

If we believe that there is increasing overlap between these categories, and all are needed for effective information use, what is the “whole thing” called?

For obvious reasons, vendors are trying to extend the terms that they are most closely associated with (e.g. Business Objects prefers BI, Hyperion prefers PM) to include the other categories. But these terms are starting to sound dated, and are firmly associated with only part of the overall need and particular users (e.g. BI = IT/query and reporting, PM = finance function/planning and budgeting). And Gartner views information management as a broad umbrella term that includes all kinds of storage and workflow technology.

So, isn’t it time for a name? Something that we can all agree on, and use to promote the value of what we all do, rather than bickering about terms?

Any nominations?