Business Intuition: A Key Input in Big Data Analytics

I’ve sometimes been asked why it is that there’s so much more emphasis on the involvement of the business in analytics projects, and why it’s any different from the need to involve the business in IT or software development projects.

This question is indeed a valid one. After all, no business software would serve its users well unless it was designed to meet their needs. In order to find out what those needs are the developers would need to speak to the business to understand their viewpoint and the problems they need solved.

In the case of big data analytics as well, the business needs to be very closely involved, but in a much more integral way. To start with, of course, analytics is used in the hope that it will help guide business choices and decisions, and answer questions by uncovering new insights. This may be considered to be analogous to providing business objectives and requirements to a software developer. But beyond this, the business needs to be much more involved, and perhaps integrated into the analytics team because the process of insight discovery requires a far greater depth of application of business domain knowledge and intuition at every stage than the software development process does.

This can be illustrated by considering the example of how big data stored using Hadoop needs to be cleaned and integrated. In conventional enterprise data stores and data warehouses data is captured, stored and retrieved according to predefined data models in a structure manner. Field widths are fixed, and data dictionaries describe how data should be formatted, what nomenclatures should be used, and tables are usually normalized. Finally, the data is all stored in regular columns and rows, and is identified and retrieved using primary keys, or correlated and joined using foreign keys. When data needs to be merged (using ETL, for example), a business perspective is needed to identify what data needs to be merged and how to reformat data as needed. But beyond that, the bulk of the exercise is a technical one that is taken care of by developers. Data is retrieved and merged according to keys and values.

With big data stores in Hadoop, however, the exercise is not as straightforward. Data may be available in different forms and from different sources. It may be structured or unstructured. Data sources may be social media, log files, devices, streams, or emails. There may not be any primary keys or foreign keys with which to conveniently extract particular records. Data formats for the same element (e.g., names) may vary widely. The only way to begin putting various elements and “records” of data together would be by using a business perspective. As an example, location data may not exist neatly in any particular database column named “Location”, but may actually need to be inferred or derived using indirect methods. An IP address, for example, could be used to reveal a location in some cases. The usage of certain words in text on social media or emails could be construed to have a positive connotation or a negative connotation (for example, the occurrence of the word “awful” in a sentence containing “service was awful”). Other data could be used to detect buying patterns. These patterns or intuitive correlations about how to put data elements together are not ones that a technical IT team would be able to make, but are ones that a business subject matter expert would need to think of and investigate. Once the technologists integrate the data as suggested, further validation is required by a business specialist in order to determine which “records” make sense for the purpose of analytics and which should be discarded.