The New York Times reports that a respected psychology journal is due to publish a paper purporting to show “strong evidence” for extra-sensory perception:

“A software program randomly posted a picture behind one curtain or the other — but only after the participant made a choice. Still, the participants beat chance, by 53 percent to 50 percent, at least when the photos being posted were erotic ones. They did not do better than chance on negative or neutral photos.”

Crucially, no “topflight statisticians” were part of the peer review. When I was at university, struggling to use a sophisticated statistics package on a mainframe as part of my econometrics degree, I dreamed of having a program that would just cruise through all the possible combinations of variables, and tell me which ones were correlated. That ability now exists, but the danger is that few people realize how much higher the bar must be set for a result to be deemed significant in such circumstances.

Given a large enough set of random numbers, you will always be able to find a “significant” relationship – especially if that’s exactly what you’re looking and hoping for.

To me, the experiment above sounds like it may have this problem – for example, if there were lots of different categories of photos, and the “significant” relationship was cherry-picked from the available results. And even if the level of significance has indeed been increased to take account of this, the result could still be random (if there’s a choice between changing everything we know about science and it being a fluke result, I’m going with the latter).

In science, thankfully, it’s easy for somebody else to repeat the experiment and validate the correlation, ideally before a respected journal makes a fool of itself (although I suspect they’re simply making a calculated bid for more publicity, and it’s working very successfully).

In business, it’s much harder to know if your “results” are valid. The same problem exists – people are looking for a certain type of result, and keep running the numbers until they find something that looks like a relationship: “Look! Customer satisfaction is correlated with their age!” . But it’s much harder to “rerun the experiment”, and businesses don’t always have/take the time to check their results.

Despite having worked in BI for over twenty years (or maybe because of it), I’m deeply distrustful of most corporate analytics. I believe business analytics is essential, but that it’s also essential to assume that any relationship you find is a working hypothesis, to be validated through further analysis (e.g. correlation is not causation), and expert discussion (as with peer-reviewed science papers, the best way to deal with potential analysis problems is greater transparency — social BI technologies like Streamwork are becoming increasingly important).

[Update: there's a great New Yorker Article that talks about the issues of finding, and replicating, significant results in studies -- much of accepted science may be false?]

Google has announced Google Refine 2.0, “a power tool for data wranglers”. It’s an open-source tool for cleaning and enhancing “messy” data sets, including cleaning up inconsistencies, transforming them from one format into another, and extending them with new data from external web services or other databases.

How does it shape up from a corporate point of view? Is this another step for Google into enterprise information management?

The new version includes:

“new extensions architecture, a reconciliation framework for linking records to other databases (like Freebase), and a ton of new transformation commands and expressions.”

The tool runs on your desktop (even though you’re accessing through a browser), so you don’t have to worry about data security.

Here are some videos about the new product. First, an introduction to how you can use the tool to do basic manual cleansing of data:

• Converting various badly-entered variants of “FFP” into “Firm Fixed Price”
• Helping “cluster” groups of similar data together using heuristics
• Using expressions to change distributions using log functions
• Identifying problems (zero values, errors between millions/billions, etc.)

Next, data transformations. The tool makes it easy to convert information in a basic HTML list into a nicely formatted table, using filtering and an expression language:

• Isolating certain rows
• Simple conversions, e.g. removing bolding
• Extracting part of the values to a new column
• Splitting existing values into new columns

The extractions used then be exported into a standard like JSON, and used to convert similarly-formatted datasets.

Finally, you can use Google Refine to augment your data with data from web services and how to link your data with databases such as Freebase:

• Calling web services to add geo-coding to address information
• Using Google’s language detection service to identify the language of different values
• Doing database joins with external data sources (Google calls this “reconciliation”)
• Freebase has a nice service that will automatically figure out what type of values you have in your data (e.g. movie names), and matches them to appropriate records, and offers you options in case of ambiguity (e.g. there are several movies containing the word “Terminator”)
• Once you have a match, you can choose from other fields (such as the movie release date, etc.)

Overall, it looks like Google Refine 2.0 is great free option for correcting and correlating small, real-world messy data sets from the web — but you have to be a JSON ninja to get the most out of it.

It’s clear that you could already use this in a corporate context — there are many one-off projects that require manual collection of information from different sources. And organizations suffer from just the same types of information fragmentation as the general web.

With a more user-friendly interface and (presumably) more robust scalability, this type of tool could be of great benefit for people trying to pragmatically cobble together  information from various data sources (a few spreadsheets, an external web site, the corporate data warehouse, etc.)

It’s not clear that corporate data is or ever will be a priority for Google — the entire enterprise information management market is a rounding error compared to their advertising empire. In the meantime, anything that can help people get the data they need, cheaply, should be welcomed.

This type of solution will never replace the need for a robust enterprise information platform, but the need for “messy” solutions to answer real-world business questions is a frequently-underestimated need in business analytics deployments, and Google Refine 2.0 looks like a great tool to add to the workbench.

Looking for some free training in these difficult economic times?

Here’s a long, long list (OK, so there are “only” 96 of them) of free web seminars courtesy of the SDN BusinessObjects Web Seminar Series and Jason Cao. They cover almost every aspect of the BusinessObjects Intelligence Platform.

And if you don’t find what you’re looking for, you can register to be notified by email of upcoming BusinessObjects webinars and suggest topics for future webinars.

Recent and Upcoming Web Seminars

Archived web seminars:

• Dashboards and Data Visualization
• Information Management
• Innovations
• Intelligence Platform
• Query, Reporting, Analysis
• Search and Navigation
• Software Development Kits (SDKs)

Dashboards and Data Visualization

Information Management

Innovations

Intelligence Platform

Query, Reporting, Analysis

Search and Navigation

Software Development Kits (SDKs)

The research also reveals that nobody knows who is supposed to be responsible for maintaining the accuracy of data. In 31% of the companies questioned, this role is held by the IT department – but IT organizations typically don’t have much control over the business processes that lead to poor data in the first place.

More encouragingly, 25% place the responsibility on a cross-functional team, and other responses included finance (10%), manufacturing (6%), sales (6%) and marketing (3%).

And sadly: in 10% of organizations, nobody is in charge.

Fixing data is essential – but hard

The best dashboards and reports in the world are useless if you can’t trust the data – you’re just putting lipstick on a pig.

Here are some of the key best practices for improving data quality:

Investigate. First monitor the existing data quality (data profiling products like SAP BusinessObjects Data Insight can be a big help here). Follow the data chain to the final users, and find all the examples and anecdotes you can about the business problems poor data quality has caused. Find examples of risks that have resulted from bad data, or any news articles about industry or competitor problems in this area. Call it data governance, and see if you can ride existing governance and compliance processes inside the organization.

Get people to care. This is the hardest part. You have to get people to feel the pain. IT organizations have a tendency to hide bad data, since they rightfully worry that it will lower the credibility of the reporting systems. But unless the data is shared, nobody will be aware of the problem, and nothing will be done (and they’ll blame IT). The trick is to provide the data, but make sure it’s clearly labeled as suspect, and make sure that there’s a link to more information about why the data is bad.

It’s all about money. Nobody cares about bad data – they care about what it’s doing to profits. Put a dollar amount on the problem. It doesn’t have to be anything complex initially – just do back-of-the-envelope calculations to see if it’s worth doing something about it. If it is, share your calculations with finance and other teams – they’ll be happy to point out your mistakes, and it might get them thinking and talking.

Create a team of data stewards. Business people think data quality is an IT problem, while IT people know better. Get a group of people together to fix the issues. Take care to set expectations correctly – it will be more complex and take longer than anybody expects. Start with the “slow, fat rabbits’” – easily fixed problems – and heavily publicize the benefits. Use the goodwill generated to tackle the harder problems.

Stop bad data getting in. It’s much cheaper to stop bad data from getting into your systems than it is to clean it up afterwards. Invest in a “data quality firewall” that checks for bad or duplicate data as it’s being entered into your operational systems (for example, SAP BusinessObjects Data Quality integrates tightly with SAP and other operational systems).

Never stop cleansing. Data degrades over time (especially customer data). You need a long-term approach to detecting, monitoring, and fixing data quality, and it needs to be made the clear responsibility of an internal team, such as a BI competency center.

More required reading

Here are some posts that might cheer you up before you tackle your data quality issues:

• Data Quality and the Art of Despair?
• Why Data Quality is Important?
• Data Quality and Bandit Sheep?

___________________

Brushes image by Bright_Tai

That’s nothing… There’s a country that I can’t name (but they play rugby and have lots of sheep). The national police force wanted a “single view of the criminal”, and so created a central data warehouse from a variety of operational and legacy systems. The ended up with a database of 4.5 million names.

This was a problem, since the population of the country is only four million. So, either everybody in the country is a criminal AND they have half a million bandit sheep…

…or they have a data quality problem.

Here, Dr. E.L. Kersten uses a mailing list debacle to illustrate his Demotivators© organizational storytelling techniques.

Remember — it could be worse…

Yet another acquisition: Business Objects purchases FUZZY! Informatik, based near Stuttgart, Germany, terms not disclosed

The important:

• A “challenger” in Gartner’s 2007 data quality magic quadrant
• Adds European address databases to Business Objects’ existing portfolio and improves European customer base/sales expertise/coverage
• Vertical solutions for Postal, Financial services, and Telco
• SOA-enabled, which should make integration easy

The trivia:

• Web site includes a great data quality quote attributed to Albert Einstein, that I’ve not seen elsewhere, and couldn’t find on Google: “it’s easier to solve data quality problems than live with them” (or, in German: “Es is leichter, Datenqualitätsprobleme zu losen, als mit ihnen zu leben”)
• UPDATE: many thanks to Vincent McBurney. I can’t believe I missed the baby name picker based on sounds
• Also, two quotes of why FUZZY Logic is important: “It’s better to be nearly right than completely wrong”, and “Tomorrow’s software must also be able to find “Pritti Wummen”. 

• Information display: Google makes it easier to show data on their maps.
  
• Data cleansing: Google submits a patent for “online data verification of listing data”. According to Intelligent Enterprise:

“The application describes a system for verifying listing information submitted by users, such as a merchant might enter when providing data to the Google Local Business Center about his or her business.”

• Information feeds: InformationWeek talks about how Twitter can be used to provide real-time information feeds:

“Twitter users send and read messages using a variety of channels: The Web, SMS, instant messaging, and RSS. The service also has an API for building third-party applications, which are springing up like weeds. These include applications to provide weather reports, tell the status of various lines on the London Underground, and provide earthquake information for Silicon Valley.”

• Data storage and linking: Freebase‘s contributors are:

“collecting data from all over the internet to build a massive, collaboratively-edited database of cross-linked data.”

• Collaboration: Trampoline Systems aims to:

“analyse data from enterprise information systems such as email servers and instant messenger tools to map social networks, information flows and collaborations throughout the enterprise.”

[more from vecosys]

More seriously, data quality is a real issue for today’s organizations. As Andy Bitterer of Gartner puts it, “there isn’t an organization on the planet that doesn’t have a data quality issue”. But most organizations don’t know how bad the problem is or where to start. An excellent first step is to invest in a data profiling tool such as Business Objects’ Data Insight.