Most successful organizations treat data-driven decision making as a primary objective and pursue it with religious zeal. However, data-driven decision making, the steps leading to it and how AI is changing it are not well-defined.
In an AI-first company, data-driven decision making means
- Strategic decisions are made by a diverse group including executives that rely on a sufficiently comprehensive set of information. By definition, a bad strategic decision should be important enough to lead to failure of the company.
- Most operational decisions are handled by continuously learning machine learning models which produce explainable decisions. Operational decisions are frequent (once a week or more frequent) and not critical (a single mistake is unlikely to lead to failure of the company).
- Operational decisions which can not be automated with good accuracy are delegated to humans.
- If data is lacking, opinion based decisions are made.
- If data exists and has been analyzed, decision maker relies on analysis.
- If data exists but is not analyzed yet, cost of analysis determines whether an opinion or data based decision will be made.
Before settling on this framework for decision making for modern corporations, we need to identify how we can evaluate different decision making models. However, if you like you can directly skip to the sections that interest you:
Why is it important?
Even though it is obvious that data-driven decision making is important, it would be a shame if we were not being data-driven about proving it. Though quantitative data about data-driven decision making is hard to come by, there is significant evidence that
- volume of data generated is increasing. Every day we create 2.5 quintillion bytes of data according to Domo’s analysis in 2017. 90% of the data in the world today has been created in the last two years.
- companies are investing more in data and machine learning technology to get insights from that data.
A relevant survey, was conducted by New Vantage Partners about Big Data investments of companies. Results highlight that businesses are trying to get an edge over each other by implementing Big Data across the company. Here are the figures in the survey:
- Percentage of companies investing more than $500M in Big Data/AI tech has increased from 13% (2018) to 21.1% (2019).
- Percentage of companies investing between $50M and $500M has increased from 27% (2018) to 34% (2019).
- Percentage of companies investing less than $50M has decreased from 60% (2018) to 45% (2019).
How to assess different decision making models?
Decision making has 3 KPIs: Speed, quality, auditability. Lack of any of these can lead to criticial failures.
- Slow decision making cripples organizations and allows competitors to gain market share.
- Bad decisions lead to failure. A bad decision, no matter how well executed, is unlikely to generate value. For example, Motorola’s decision to continue investing in Iridium satellites which cost billions of dollars resulted in a big write-off for the company. Great execution could not change the outcome as mobile communication was providing a cheaper and higher quality alternative to satellite phones by the time Iridium phones were launched.
- If decisions can not be audited, they are hard to improve. Decision quality is difficult to measure at the time of the decision but can be estimated later by considering other alternatives and current market conditions. Such postmortem analysis should lead to better decisions
What are different decision making models?
Decision making models starting from the lowest level of sophistication are:
1- Opinion based
While it provides fast decisions, decision quality can be low which is dangerous for strategic decisions. However, it can be applied for fast decisions making in cases where:
- High quality data is not available
- Cost of analysis is more than the value of a good decision
2- Driven by limited amount of data
Organizations with data quality or availability issues are likely to rely on limited amounts of data to make decisions. This is potentially the most dangerous decision making model as decision makers could use limited data to support their opinions, leading to widely supported but opinion based decisions.
A significant share of consulting projects fall under this category where consultants could be incentivized to find limited data points to support opinion based hypotheses.
Decision quality can be better than opinion based decision making if decision makers are open to new information, take into account psychological biases, vary their approach to making prediction (inside view vs outside view) and look for answers rather than searching for data to support their opinion.
3- Data-driven decision making via manual analysis
For this model, data needs to be available, high quality and there needs to be consensus about the correctness of data. It can be preferable for important decisions where timing is not critical and where cost of automation is prohibitive compared to its benefits.
4- Automated data-driven decision making
Data is available and high quality and there is consensus about the correctness of data and it has been processed so decisions are made automatically. Based on outcomes, the decision making model continuously improves itself over time. However, audit is challenging when reasons for decisions are not provided.
5- Automated data-driven explainable decision making
In addition to automated data-driven decision making, explainable machine learning models are used to make decisions. Explanations help audit decisions and continuously improve the underlying model via manual interventions on top of automated continuous learning.
Why I wrote this?
I spent most of my professional career at McKinsey where data-driven decision making is one of the core principles. It was a great principle but quite challenging to achieve as a consultant without access to internal data. Given my current experience with AI, I wanted to think how we can better define data driven decision making.