Google has announced Google Refine 2.0, “a power tool for data wranglers”. It’s an open-source tool for cleaning and enhancing “messy” data sets, including cleaning up inconsistencies, transforming them from one format into another, and extending them with new data from external web services or other databases.

How does it shape up from a corporate point of view? Is this another step for Google into enterprise information management?

The new version includes:

“new extensions architecture, a reconciliation framework for linking records to other databases (like Freebase), and a ton of new transformation commands and expressions.”

The tool runs on your desktop (even though you’re accessing through a browser), so you don’t have to worry about data security.

Here are some videos about the new product. First, an introduction to how you can use the tool to do basic manual cleansing of data:

• Converting various badly-entered variants of “FFP” into “Firm Fixed Price”
• Helping “cluster” groups of similar data together using heuristics
• Using expressions to change distributions using log functions
• Identifying problems (zero values, errors between millions/billions, etc.)

Next, data transformations. The tool makes it easy to convert information in a basic HTML list into a nicely formatted table, using filtering and an expression language:

• Isolating certain rows
• Simple conversions, e.g. removing bolding
• Extracting part of the values to a new column
• Splitting existing values into new columns

The extractions used then be exported into a standard like JSON, and used to convert similarly-formatted datasets.

Finally, you can use Google Refine to augment your data with data from web services and how to link your data with databases such as Freebase:

• Calling web services to add geo-coding to address information
• Using Google’s language detection service to identify the language of different values
• Doing database joins with external data sources (Google calls this “reconciliation”)
• Freebase has a nice service that will automatically figure out what type of values you have in your data (e.g. movie names), and matches them to appropriate records, and offers you options in case of ambiguity (e.g. there are several movies containing the word “Terminator”)
• Once you have a match, you can choose from other fields (such as the movie release date, etc.)

Overall, it looks like Google Refine 2.0 is great free option for correcting and correlating small, real-world messy data sets from the web — but you have to be a JSON ninja to get the most out of it.

It’s clear that you could already use this in a corporate context — there are many one-off projects that require manual collection of information from different sources. And organizations suffer from just the same types of information fragmentation as the general web.

With a more user-friendly interface and (presumably) more robust scalability, this type of tool could be of great benefit for people trying to pragmatically cobble together  information from various data sources (a few spreadsheets, an external web site, the corporate data warehouse, etc.)

It’s not clear that corporate data is or ever will be a priority for Google — the entire enterprise information management market is a rounding error compared to their advertising empire. In the meantime, anything that can help people get the data they need, cheaply, should be welcomed.

This type of solution will never replace the need for a robust enterprise information platform, but the need for “messy” solutions to answer real-world business questions is a frequently-underestimated need in business analytics deployments, and Google Refine 2.0 looks like a great tool to add to the workbench.

[UPDATE: See this post for the soon-to-be released prototype from the SAP BusinessObjects Innovation Center]

Here’s an example of “augmented corporate reality” based on the Layar platform and the SAP BusinessObjects ondemand.com platform, running on a Google Nexus One Android phone.

These are all screenshots taken from a working proof of concept, shown as an example of the type of functionality business users might have access to in the future — not with any promise of such a product in the future.

Open the Layar application, look at the list of available “layers”, and choose the SAP Web 2.0 Augmented Corporate Reality Example
The application opens the camera’s phone, and connects to a business intelligence server to get information about the customer businesses closest to the current location. This information is shown in the small radar screen in the top right, and an icon for each customer is shown on the screen. As the phone moves, the icons appear to stay in one place, hovering over the physical location of that customer’s building. Each icon indicates the customer’s sales performance (the arrow direction shows the trend, the color indicates whether or not sales have reached my target or not). The size of the icon indicates roughly how near or far the office is. The application automatically highlights the location closest to the center of the display, and shows more information about that business (name, distance, photo, current and previous sales, and sales target)
Users can click on the selector in the top left to see the customers as a list, ordered by distance from the current location (realistically, this is likely to be the easiest, most popular location-aware information use)
Alternatively, we can see the customers, and their performance icons, shown overlaid on a map.
When the user clicks on one of the icons, they see more data about that business, and can click on the “Get Data” button to drill in for more information.
This takes them to a live report hosted on the SAP BusinessObjects on-demand business intelligence environment…
…where they can drill into the numbers, open other reports and charts, etc.

We hope to open up the prototype for use by others, with the ability to add other example locations, etc.

What could businesses really do with this? Well, maybe you’re a branch manager, and you want to get the latest sales figures as you visit the branch:

Or a factory manager who wants to see the maintenance records for a particular machine:

Or you’d like to compare performance of goods displayed inside the store with the goods stored in the shop window:

What you can do with this technology is currently restricted by the location services: realistically, it works well for identifying building locations outdoors using GPS, but is not yet precise enough for most indoor use. However, this is changing fast as vendors offer new location services, based on cell-phone, wifi, and other forms of signal triangulation.

How far could this go? Well… take a look at this video:

SAP today announced new collaboration agreements that allow SAP Business ByDesign users to more easily leverage web services from third-party providers, including:

Business and investment news. Users can access investment research and other data from Hoover’s, Morningstar, and Business Wire, Google News, and Google Finance, and combine it with their own information about customers and suppliers.

Mapping and address services. Users can map and route customers, suppliers and other contacts using services from Google Maps, MapQuest.com, Navteq/Map24, and Falk online.

Search. Users can leverage Google Web Search, and Google Product Search, and GoYellow.de, a German service that allows reverse phone lookup functionality.

Further web services are planned to be introduced in the next release of Business By Design in Q3 of 2009.

Frustratingly, no information was available about the extent of integration with these services, and how customers could proceed. As Chris Kanaracus reported in his article SAP adds Web Services to Business by Design:

It wasn’t immediately clear how deep the integrations are between the services and BBD. This is a key point, said Forrester Research analyst Ray Wang.

Just being able to call out to Google Maps from within the ERP suite has limited value, Wang said. But if the system could use sales data contained in BBD to create a “heat map” showing the hottest territories, “that would be pretty cool,” he said.

At some point more information may be available on Web Services area of the Business ByDesign marketplace (registration required). For now, it contains roughly the information as the press release.

Google has posted videos of all the sessions at the recent Google I/O 2009 developer’s conference.

SAP attended as part of the session on “OpenSocial in the Enterprise”). Overall, the session was a little disorganized and meandering, but it did end up giving an overview of what companies like IBM, Oracle, SalesForce.com, and Atlassian are doing to leverage social networking technologies such as OpenSocial within organizations:

• Mark Weitzel gave a demo of the IBM Mashup Center proof of concept, allowing you to create mashup pages, using messaging between different sources (based on OpenAjax, but can integrate with OpenSocial) (link directly to this demo)
• Rich Manalang from Oracle gave a demonstration of Oracle’s internal collaboration system, called “Oracle Connect”. (link directly to this demo)
• Dave Carroll of Salesforce.com showed how they have been working with customers to add collaboration to the CRM experience (link directly to this demo)
• Mike Cannon-Brookes of Atlassian showed how OpenSocial could be used to link enterprise applications, and between applications and gmail. For more, see http://atlassian.com/opensocial (link directly to this demo)

The highlight of the session (OK, so I’m biased) was a demonstration by John Mayerhofer, VP, Standards Strategy Group of SAP, showing the Social Network Analyzer technology (see below, or link directly to this demo)

Unfortunately, not much time was left for Q&A session. The most interesting question was about “What are the barriers to social systems in the enterprise?” and the answers included a mix of technical and social concerns:

• Security and identity were mentioned as special problems.
• There was some frustration than social networking APIs don’t take into account enterprise needs.
• Exposure to the consumer social networks actually leaves a negative impression, people fear that it’s just about socializing, rather than collaborating.
• Everybody agreed that it was early days
• The session ended with a hopelessly vague question/comment about needing to “liberate users” rather “concentrating on the enterprise”.

There is also information available on how to make your public information more easily available.

How long before (a) you’ll be able to do this within an organization using OneBox? and (b) this is extended beyond the simple public data sources?

See “Find public statistical data through Google” and this post for more.

It’s a great idea, and there’s surely a great future for this technology. But there are currently some issues with relevance — e.g. I put in the search term “productivity”, and chose the first chart it offered me: the mean number of chickens in a flock, and the % of respondents?!…  

The other problem is that the site uses Adobe’s SVG viewer to provide interactivity — but Adobe is pulling support for the product at the end of this year…

Of course the site is still in Beta. I wish them all the best of luck, but can’t help thinking that it’s part of a solution rather than an end-goal in itself.

At some point, will Google hoover up the different BI 2.0 offerings, integrate them with their existing analytics tools and gapminder? The result would be a compelling consumer-oriented BI solution which could then be extended to enterprise users…

…Google breaks down mobile users into three behavior groups:

A. “Repetitive now”
B. “Bored now”
C. “Urgent now”

The “repetitive now” user is someone checking for the same piece of information over and over again, like checking the same stock quotes or weather. Google uses cookies to help cater to mobile users who check and recheck the same data points.

The “bored now” are users who have time on their hands. People on trains or waiting in airports or sitting in cafes. Mobile users in this behavior group look a lot more like casual Web surfers, but mobile phones don’t offer the robust user input of a desktop, so the applications have to be tailored.

The “urgent now” is a request to find something specific fast, like the location of a bakery or directions to the airport. Since a lot of these questions are location-aware, Google tries to build location into the mobile versions of these queries.

The categories work well for corporate use of BI. “Repetitive now” is clearly equivalent to production reporting and dashboards. “Urgent now” describes a power user responding to an executive request. “Bored now” doesn’t quite translate, but it could be considered the equivalent of an analyst digging for trends when not busy responding to urgent requests.

Just as personal search has ushered in new interest and investment in corporate search and text analytics, I believe that personal and mobile BI trends will have a great influence on corporate BI in the future.

• Information display: Google makes it easier to show data on their maps.
  
• Data cleansing: Google submits a patent for “online data verification of listing data”. According to Intelligent Enterprise:

“The application describes a system for verifying listing information submitted by users, such as a merchant might enter when providing data to the Google Local Business Center about his or her business.”

• Information feeds: InformationWeek talks about how Twitter can be used to provide real-time information feeds:

“Twitter users send and read messages using a variety of channels: The Web, SMS, instant messaging, and RSS. The service also has an API for building third-party applications, which are springing up like weeds. These include applications to provide weather reports, tell the status of various lines on the London Underground, and provide earthquake information for Silicon Valley.”

• Data storage and linking: Freebase‘s contributors are:

“collecting data from all over the internet to build a massive, collaboratively-edited database of cross-linked data.”

• Collaboration: Trampoline Systems aims to:

“analyse data from enterprise information systems such as email servers and instant messenger tools to map social networks, information flows and collaborations throughout the enterprise.”

[more from vecosys]

“The generating of the dictionary data may also include producing data related to the information indicative of the user location. The generated dictionary data may be associated with places near the user location. Also, the generated dictionary data may be associated with common query data from users near the user location, and may be provided to the remote device in response to a request from the remote device.”

In theory, this could include “consumer BI” queries such as:

• “the three closest restaurants with a Zagat’s rating of over 16″
• “the closest gas station with below-average price per gallon”
• “a dentist within 10 miles that has no complaints on the web”

From a business point of view, an area sales manager could automatically be provided with the latest figures for a customer or sales office as soon as they got within a certain distance of that location.

[UPDATE: it looks like this is happening. More on Google's plans in this article]