Risk Management in Analytics Projects

Inadequate data. Statistical analysis is often done on a sample of data rather than on a 100% complete data population because it is often neither practical nor possible to have data from the whole population. In such cases statistical results are expressed along with a calculated margin of error, which is fine. The larger the sample size relative to the total population, the greater the confidence in the result, ie, the lower the potential margin of error. But what if the real population size is not really known? Assumptions can be made, but how reliable are these assumptions? What are they based on? It may be worth asking the question and ascertaining from the right sources that the total population size is as accurate as it can be.

Lack of awareness about future change. This applies especially to predictive analysis, where data from the past is used to make probabilistic predictions about the future. When making such predictions the biggest risk is that the future environment could be different, and therefore the past is not enough of a determinant of the future. This is actually a fundamental thumb rule in the world of financial investment, where the utilization of publicly unavailable knowledge to make gains is illegal in most countries. However this is not the case in other areas of making predictions about the future, and so while the past can be used to produce good predictions, wherever possible, any variables that could change value in future should be considered to see if they would make any change to the statistical prediction.

Data Quality. Data quality could be an issue when testing of data integration and data cleansing techniques is done on the basis of sample testing, especially when Big Data is involved. While the techniques may produce good data in development and test samples, what is the confidence that the sample represents the entire data population well enough? Again, this is not something that requires a 100% check of the data population, but when it results in significant skews or outliers it’s always worth asking the question and then going back to double check the quality of the data points leading to such results.

Not enough team input included. Business analytics can be an expensive investment, given the cost of talent and also the amount of time that may be needed before benefits are realized. At the same time, the value of soft power in analytics cannot be underestimated. Getting the right data together, analysing models correctly and asking the right questions of the output requires as much creativity, business expertise and experience as possible, and therefore even if the core analytics team is small it helps if they engage with as many other colleagues as possible to get additional perspectives on their line of thinking.

Biased interpretations. Sometimes, though, the power of experience and gut feel may also come in the way, and that’s when there has to be that interesting discussion and debate between the statistician who is detached from the business, and the business expert who has so much knowledge of the domain that they may only expect to see validation of what they guess to be right. It’s a risky situation when either one side forces through a view that is potentially biased because it ignores either the data or past experience in some way, either consciously or inadvertently.

Group think. Group think is a twist on the biased interpretation issue. It refers to the phenomenon where those who don’t really have an opinion or don’t wish to voice an opinion defer to another opinion that seems more credible for whatever reason, and go along with it without really having any basis for doing so. As an example, in a group of three persons, if one states an opinion and the other two go along with it without any real conviction or basis other than accepting the authoritative knowledge of that person, on paper the result would be that three people voiced the same opinion. Group think is a common phenomenon and since the risks of it happening can be expensive it is something that any analytics leader should be aware of.