The New York Times reports that a respected psychology journal is due to publish a paper purporting to show “strong evidence” for extra-sensory perception:

“A software program randomly posted a picture behind one curtain or the other — but only after the participant made a choice. Still, the participants beat chance, by 53 percent to 50 percent, at least when the photos being posted were erotic ones. They did not do better than chance on negative or neutral photos.”

Crucially, no “topflight statisticians” were part of the peer review. When I was at university, struggling to use a sophisticated statistics package on a mainframe as part of my econometrics degree, I dreamed of having a program that would just cruise through all the possible combinations of variables, and tell me which ones were correlated. That ability now exists, but the danger is that few people realize how much higher the bar must be set for a result to be deemed significant in such circumstances.

Given a large enough set of random numbers, you will always be able to find a “significant” relationship – especially if that’s exactly what you’re looking and hoping for.

To me, the experiment above sounds like it may have this problem – for example, if there were lots of different categories of photos, and the “significant” relationship was cherry-picked from the available results. And even if the level of significance has indeed been increased to take account of this, the result could still be random (if there’s a choice between changing everything we know about science and it being a fluke result, I’m going with the latter).

In science, thankfully, it’s easy for somebody else to repeat the experiment and validate the correlation, ideally before a respected journal makes a fool of itself (although I suspect they’re simply making a calculated bid for more publicity, and it’s working very successfully).

In business, it’s much harder to know if your “results” are valid. The same problem exists – people are looking for a certain type of result, and keep running the numbers until they find something that looks like a relationship: “Look! Customer satisfaction is correlated with their age!” . But it’s much harder to “rerun the experiment”, and businesses don’t always have/take the time to check their results.

Despite having worked in BI for over twenty years (or maybe because of it), I’m deeply distrustful of most corporate analytics. I believe business analytics is essential, but that it’s also essential to assume that any relationship you find is a working hypothesis, to be validated through further analysis (e.g. correlation is not causation), and expert discussion (as with peer-reviewed science papers, the best way to deal with potential analysis problems is greater transparency — social BI technologies like Streamwork are becoming increasingly important).

[Update: there's a great New Yorker Article that talks about the issues of finding, and replicating, significant results in studies -- much of accepted science may be false?]

Today, most corporate decision-making happens in meetings and conference calls, with little support from technology. As a process, it functions, but most of us would be hard-pressed to call it optimized.

in many ways, corporate decision-making resembles the manual, time-consuming operational processes that have been replaced by automated, connected business applications.  Isn’t it time that we applied some of the lessons we’ve learned while implementing ERP to the executive boardroom?

The payoff is huge: organizations make thousands of decisions every day. A little better data, and a little better analysis behind every decision can make a big difference.

Just as most technology discussions don’t talk about the decision-making process, most discussions about decision-making don’t mention technology. Here’s a case in point: SAP’s Don Bulmer recently posted on the subject of Leadership: Making the Tough Decisions. It was a great article, but we could also talk about the technology behind the principles he laid out.

So here’s Don’s three basic steps to making tough decisions, followed by my comments on how technology could – and should – help more in the future.

Understand what is the core issue, and separate facts from opinions

First, it is important to understand the ‘core issue’ associated with the business challenge that you are dealing with and gather as many facts and insights surrounding the issue as possible. At this stage in the process, it is important to separate ‘facts’ from‘opinions.’

As a leader, you are often presented with ‘facts’ that are presented to support a specific argument or point of view – which by default make them adverse. This is not to say that these adverse or ‘position-based’ facts are not valid, but they seldom represent a fully informed view of the core issue that needs to be addressed.

Any decision you make is only as good as the information you base it on. Accurate, transparent decision-making requires a strategic approach to information across the organization. If everybody comes to the meeting with their own spreadsheets and numbers defending their point of view, you’ll end up spending more time arguing about the data than about the decision itself. A “single view of the truth” across the organization, with consistent, reliable information, makes it much easier to separate fact from opinion.

Look at the decision from all angles and perspectives

Second, it is important to look at your decision options from all angles and perspectives. Many tough business decisions will have a broad effect on your business which might include: brand; reputation; relationships with employees, customers, partners, governments, etc. It is very common to overlook the total impact that a tough decision will have on your business. This is why it is important to consider many options – as you weigh the impact and trade-offs that most important decisions will require.

As a leader, this is where intuition and experience play a big role.

No computer can help you figure out all the angles of a decision, but technology can help. For example, profitability and costing analysis can help ensure that everybody involved in a decision understands the real drivers of the business: what is value-added activity, and what is not? What should you be spending time on, and what is less important? And predictive analysis can help determine what factors have historically had the most influence on the numbers at hand, and determine what might happen in the future.

Ask lots of questions of lots of people

Third, it is important to ask a lot of questions. This will help you to understand the options on the table as you make sense of the facts and sift through potential agendas/motives of people involved. Asking the right questions also helps you to measure the risk and impact that your decision will have on your business and with key relationships.

You may even consider speaking with trusted representatives from affected internal or external groups (customers, employees, partners,etc.) to get their perspective. This interaction will also help secure support and buy-in (by all parties) of your final decision once you are ready to communicate.

The ‘tougher’ the decision the more challenges you can expect to face as you communicate and implement. You will need to be prepared to help affected parties understand the rationale behind the decision and even share some of the alternatives considered and why you chose a different path.

This is the key area that technology is poised to help more with in the future. Today, there’s relatively little support within Business Intelligence industry for the process of gathering and sifting information from different people involved in decisions. Interesting technology exists today, but is not yet integrated into a whole solution. For example:

• Antivia has done some very interesting work in the area of “BI 2.0” and letting people collaborate around reports and information within a BI platform
• Panonica had some interesting collaboration software, with tight links to BI, back in the early 2000s, but was clearly too ahead of its time, and has since been discontinued
• ExpertChoice and others have supported collaborative decision-making, but without the BI angle.

The next generation of Business Intelligence will combine the best of these different technology approaches to better support decision-making, not just information-gathering. For more information about how technology can help with decision-making, see The 5 Ingredients of Good Decision-Making, based on some research done by the Economist Intelligence Unit.

Other posts related to decisions:

• Bad Decisions: Just Blame Evolution?
• More Milk Please, Ermintrude! A Classic Decision Trap?
• Is There Such a Thing as Good Gut Decisions?
• Will Computers Ever Help With Decisions? (No)
• Intestine-Based Decision-Making

The results of a survey analyzing the current status of corporate decision-making were announced today, called “In search of clarity: unraveling the complexities of executive decision-making”

The survey of 154 global C-level executives, conducted by the Economist Intelligence Unit (EIU) and commissioned by Business Objects, found that more than nine out of 10 corporate executives admit they are making important decisions on the basis of inadequate information. 

More than half of these senior executives are concerned that they may be making poor decisions as a result of missing information. And a quarter believe that management frequently or always gets its decisions wrong.

These are sobering statistics. We are talking about business decisions that can cost an organization millions of dollars, either through costly errors or through the failure to grasp a competitive advantage.

Despite eight out of ten respondents indicating that data is the most important factor in making decisions, ranking it much higher than the opinion of others, personal intuition, or external consultancy, the research showed that more than half of respondents said that decision-making in their organizations was largely informal and ad hoc, and the the report concludes that:

“Decision-making is at the core of all business activity, as executives set strategy and manage operations by weighing a vast array of factors to arrive at the desired balance of risk and reward. It is cause for alarm, then, that executives themselves perceive the quality of decision-making at their companies as mixed at best.”

At a time when the economists are predicting that the current credit squeeze is likely to impact the business environment, it is important to wring out every advantage amidst intense and global competitive pressures.

The simple fact is that executive decision makers are not getting the data they value and need, and most executives, like ourselves, are relying more on “gut instinct” than metrics.

But, why would gut feel override our desire for proof? For fact?

What is failing us? Is fact-based decision-making too difficult to sustain?

Five Ingredients of Good Decision-Making

The report’s authors identify five ingredients of good decision-making. Obviously, supporting good decisions requires a lot more than technology, but I believe BI can help with each of these…

• High-quality data
• Access to advanced systems and training
• Sound judgment
• Trust
• Flexibility

High-Quality Data

High-quality data is obviously a prerequisite for consistently sound decision-making. The better the data, the less time you’ll spend debating the data rather than the decisions that have to be made, and the greater your understanding of your company, your competitors and your environment, the more you can move from guesswork to making strategic choices.

Here are some of the difficulties organizations typically face getting high-quality data, and what can be done to resolve them:

• Data integration. Executive decisions typically require information from several different computer systems, and organizations frequently underestimate the integration challenges involved. Organizations typically address data integration on a project-by-project basis rather than with a more strategy approach. The result is massive redundancy and poor quality. For faster integration and lower costs over the long term, organizations should invest in a robust data integration platform.
• Data quality. There isn’t an organization on the planet that doesn’t have a data quality problem, but where do you start? The latest data quality technologies can help you answer questions such as “which systems have the worst data quality?” and “where could I get the most value from fixing poor quality data?” To improve executive decision-making, organizations should invest in systems to evaluate, monitor and fix data quality.
• Shared definitions. Effective decision-making requires a shared vocabulary. Investing in “metadata management” technology can help you define and maintain the definitions associated with your key business indicators — and reduce the number of board-room arguments.
• Timeliness. Faster decisions mean higher profits. In particular, organizations are starting to realize the big benefits that can come from investing in integrated financial systems that help them close their books faster. And rather than managing by “looking out of the rear-view mirror”, organizations are investing in predictive analytics.
• Unstructured data. Executives need to be use data from all sources, not just from certain types of databases. It’s increasingly important to be able to extract intelligence from non-structured data sources such as documents or emails.
• Benchmarking and external data. Your own systems will rarely contain all the information you need to make decisions. In order to determine your real performance, and make the right decisions, you need to be able to compare your numbers with the economy, the market, or your competitors. New “information on demand” technology is making this as simple as a click of the mouse.
• Governance, compliance, and risk. Decisions can’t be made without considering risk, and without considering your regulatory environments. Organizations should invest in risk management systems that are tightly integrated with the rest of their financial applications.
• Transparency. Information transparency is key to maintaining high data standards. Executives need to be able to see exactly where the data came from, how reliable it is, how it was defined or manipulated, and when it was last updated.

Access to Advanced Systems… and Training

Access to advanced information systems is crucial to improved decision-making, as is training in helping employees to make full use of these tools. There is no point in spending on new technology if people do not use it.

• Ease of use. Th
  e easier the technology, the more people will use it. The latest BI innovations bring information to the users’ fingertips by making it a seamless part of their existing environments, whether it’s email, a standard productivity application like Microsoft Word or Excel, or a cell phone. In the future, BI will be “ambient” – like ambient lighting, it will bring illumination without calling attention to itself.
• User adoption. The implementation of BI technology should be considered the start of a project, not the end. Better access to data doesn’t provide any benefits – that only happens when business processes change based on the new information. Training must be more than a one-off activity, and encompass all aspects of business change management.
• Standardization. Having a standard BI environment across the organization helps provide economies of scale on all aspects of training and user adoption, and makes it easier to create a community of users that can support less experienced users.
• Competency centers. Organizations should invest in a BI competency center: a team that is dedicated to making the best use of the organization’s information assets, ensuring the right trade-offs between the needs of each department/project and the company as a whole.

Sound Judgment

Decision-making processes, whether formal or not, need to leverage the strengths of human intuition. Data does not run companies; people do.

• Collaboration. Intuition is very important, but it needs safeguards — and one of the best is other people. It must be easy to share data, different interpretations of what data means, and proposed plans to improve the situation. The more information is shared, the more likely it is that bad decisions are avoided.
• Guided analysis and best practice applications. Many decisions have to be made on a regular basis by lots of different employees. Organizations should propose a consistent set of analysis steps in order to help every user make decisions as well as the best analyst.
• Links to financial planning. Business decisions have to be backed up by appropriate business changes, including budgets. Your business intelligence systems should be closely linked to your organization’s financial systems.
• Profitability analysis. Make sure that decisions are being made based on what’s important: profit. After all, there’s no point in maximizing revenue if you’re making a loss on each product.
• Sharing with the business ecosystem. It’s no longer just about your organization. You increasingly need to make decisions based on the operations of the “business ecosystem” of customers, partners, and suppliers. For example, to make sound decisions about quality you may have to share and collect warranty information from your distributors.

Trust

To gain employees’ confidence in management decisions, establishing transparency and trust is at least as essential as a good track record.

• Shared vision. The more widely information is shared across the organization, the easier it is for employees to understand and carry out executive decisions – and to provide critical front-line feedback.
• Linking strategy to execution. Nine out of ten organizations struggle to execute their strategy. One thing that can help is clear scorecards and dashboards that cascade high-level goals into key performance indicators that are tracked for teams and individuals. In particular, this helps make the inevitable tradeoffs between different decisions more explicit and transparent.
• Compensation management. People must not only understand the strategy, but understand why they should follow it. There are few things that are more likely to make employees resist and mistrust decisions than misaligned incentives. Applications exist that help organizations plan and implement optimal compensation management strategies – and track if they’re actually achieving the desired goals.

Flexibility – One Size Does Not Fit All

Approaches to decision-making, and even to the use of data, need to reflect the fact that the world is a diverse place, and one size does not always fit all.

• Standard platform. Letting every individual or group look after its own information needs is a recipe for disaster. A BI competency center should be given the mandate to make the necessary trade-offs between flexibility and standards. In today’s fast-changing business environments, flexibility is achieved most easily with a standard platform but a variety of different techniques and interfaces appropriate for different users and situations.
• Collaboration. Organizations increasingly realize that decision-making is an activity that needs to be opened up to more people in the organization. Top-down execution of fixed strategies is giving way to more flexible, collaborative approaches, and a common “information infrastructure” is an essential enabling technology.
• Service-oriented architectures. Computer systems are moving from monolithic suites to more modular, “services-oriented” architectures. As business intelligence becomes more process-oriented, organizations need the ability to easily share and consume information services that can be adapted by business users without further IT assistance.
• Independence. The more volatile the environment, the more companies need to be able to access critical information fast. Your information systems should be maintained separately from your underlying operational systems, and support any and every environment, and be ready to accept information from new systems – you never know when you might merge with another organization, for example.

Conclusion

Technology is necessary but insufficient condition for good decision-making and performance excellence. Without a solid information infrastructure, and support for financial planning, the best decisions and strategy in the world is unlikely to succeed.