Planning for Big Data – A CIO’s Handbook to the Changing Data Landscape”, by the O’Reilly Radar Team, featured a chapter by Alistair Croll, entitled “The Feedback Economy” which informs today’s discussion. Croll believes that big data will allow continuous improvement through the “feedback economy”. This is a step beyond the information economy because you are using the information that you have generated and collected as a source of information to guide you going forward. Information itself is not the greatest advantage but using that information to make your business more agile, efficient and profitable is. Croll draws on military theory to illustrate his concept of a feedback loop. It is the OODA loop, which stands for observe, orient, decide and act. This comes from military strategist John Boyd who realized that combat “consisted of observing your circumstances, orienting yourself to your enemy’s way of thinking and your environment, deciding on a course of action and then acting on it.” Croll believes that the success of OODA is in large part “the fact it’s a loop” so that the results of “earlier actions feedback into later, hopefully wiser, ones.” This should allow combatants to “get inside their opponent’s loop, outsmarting and outmaneuvering them” because the system itself learns. For the business leader this means that if your company is able to collect and analyze information better and you can act on that information faster. Croll believes one of the greatest impediments to using this OODA feedback loop is the surplus of noise in our data; that “We need to capture and analyze it well, separating the digital wheat from the digital chaff, identifying meaningful undercurrents while ignoring meaningless flotsam. To do this we need to move to more robust system to put the data into a more usable format.” Croll moves through each of the steps in how a company collects, analyzes and acts on data. The first step is data collection where the challenge is both the sheer amount of data coming in and its size. Once the data comes in it must be ingested and cleaned. If it comes into your organization in an unstructured format, you will need to cut it up and put into the correct database format for use. Croll touches on the storage component of where you place the data, whether in servers or on the cloud. A key insight from Croll is the issue of platforms, which are the frameworks used to crunch large amounts of data more quickly. His key insight is to break up the data “into chunks that can be analyzed in parallel” so the data can be considered and acted upon more quickly. Another technique he considers is “to build a pipeline of processing steps, each optimized for a particular task.” Another important component is machine learning and its importance in the data supply chain. Croll observes, “we’re trying to find signal within the noise, to discern patterns. Humans can’t find signal well by themselves. Just as astronomers use algorithms to scan the night’s sky for signals, then verify any promising anomalies themselves, so too can data analysts use machines to find interesting dimensions, groupings or patterns within the data. Machines can work at a lower signal-to-noise ratio than people.” Yet Croll correctly notes that as important as machine learning is in big data collection and analysis, there is “no substitute for human eyes and ears.” Yet for many business leaders, displaying the data is most difficult because it is not generally in a readable form. It is important to portray the data in more visual style to help convey the “dozens of independent data sources” into navigable 3D environments. Of course having all this data is of zero use unless you act on it. Big data can be used in a wide variety of decision making, from employment decisions around hiring and firing decision, to strategic planning, to risk management and compliance programs. But it does take a shift in compliance thinking to use such data. It advocates “fast, iterative learning.” Big data allows you to make a quicker assessment of the impact of measured risks. Croll ends his chapter by noting that the “big data supply chain is the organizational OODA loop.” But unlike the OODA loop, it is more than simply about the loop and plugging information as you move through it. He believes “big data is mostly about feedback”; that is, obtaining the impact of the risks you have accepted. For this to work in compliance, a company’s compliance discipline needs to both understand and “choose a course of action based upon the results, then observe what happens and use that information to collect new data or analyze things in a different way. It’s a process of continuous optimization”. The OODA loop coupled with the data that is available to you should facilitate a more agile and directed business. The feedback components allows you to make adjustments literally on the fly. If that does not meet the definition of continuous improvement, I do not know what does. [tweet_box design=”default” url=”http://wp.me/p6DnMo-3u4″ float=”none”]The OODA feedback loop allows you to make business adjustments literally on the fly. [/tweet_box]]]>