By Anthony Agresta, VP, Centrifuge Systems

http://www.ebizq.net

Published January 25th, 2010

The assessment of what went wrong in the recent Christmas Day airliner attack over Detroit is clear: we should have done a better job identifying the warning signs of a pending attack by connecting the dots across a series of data sources.

Because of the severity of the potential outcome, this incident takes on a level of scrutiny that is unparalleled. But terrorism is just one example where the ability to "connect the dots across data sources" and share the results can have tremendous benefits.

Criminal activity is climbing as individuals attempt to conceal their identity in the darkest reaches of the Internet. Countless situations such as fraud, money laundering, cyber attacks, data breaches, illegal gambling, and child pornography exist where the ability to obtain relevant, timely analytical results could prevent serious harm from happening to people and organizations.

 

The need to connect the dots is not limited to law enforcement and counter terrorism. Data analysis plays an important role in virtually every industry. Businesses, for example, need to better understand financial performance. Manufacturers need to be aware of the dependencies between orders, inventory, shipments, and delivery timetables. Pharmaceutical companies need to understand the results of new, vitally important drug trials and the resulting interactions on patients. Energy companies need to explore sources of energy, to name a few.

 

Unfortunately, traditional business intelligence (BI) and data mining applications provide only a modicum of analysis. Imagine what the result would be if the analyst could visualize disparate data in rich pictures and draw linkages between people, places, and events? How much better could the result be if the same analyst could quickly connect to new data sources and uncover meaningful insight in minutes? How beneficial would it be if results could be shared in real time with someone halfway around the world?

 

Traditional BI solutions do not allow for this level of exploration. Rigid in nature, they can be hard to install and difficult to learn. Most existing BI tools rely on underlying data models used in the analysis. If the data needed for the analysis is not present, long cycle times and missed opportunities follow.

 

Further, they lack the highly interactive data visualization environment needed to rapidly uncover hidden relationships in data, fail to provide the ability to integrate data and share results, and isolate the analyst from other key players that need insights to help guide decisions.

 

Emerging technologies

Three emerging innovations in BI that can improve data analysis are interactive visualization, unified data views and collaborative analysis. These technologies comprise the pillars of Interactive Analytics, a human-centric approach to data analysis.

Interactive Analytics holds great promise for quickly and effectively detecting relevant insights, and is based on highly interactive visualizations that allow analysts to rapidly identify and comprehend unobvious patterns within the data. These visualizations include relationship or network graphs for link analysis, summary charts and heat maps for quantitative analysis, timelines for temporal analysis, and maps for geospatial context.

 

Interactive visualization

Most analysis tools require one to know what one is looking for in advance, such as: "What is my revenue by region?" or "How many customers do I have?" When one needs to explore the expanse of data, the tool falls short. Yet this is precisely what businesses must do today -- discover the unknown.

 

Information visualization addresses this issue by using visual metaphors to enhance the ability to detect patterns in data. Interactive visualization builds on this by freeing the analyst to interact directly with the visualizations, ask open-ended questions, and pursue a line of inquiry. When something relevant is found, inferences are drawn almost instantly, allowing the analyst to work at the speed of the human brain.

 

One type of visualization making its way into the mainstream is the relationship graph. Also called node-and-link diagrams, relationship graphs fall under the science of link analysis, which is used to discover and understand relationships between seemingly unrelated entities. Imagine, for example, looking at reams of tabular data relating to flight and housing records of foreign visitors to the country. Now, look at the same data in a relationship graph. It quickly becomes obvious that multiple people arriving on different flights are going to the same address.

 

The ability to switch between multiple, integrated views of a data set is a powerful paradigm for visual analysis. Integrated views allow one to "shift the lens," such as moving from a chart that summarizes the magnitude of the problem to a time-line analysis that can show intervals between events. This lets the analyst visualize the data in different forms, each of which reveals something new.

 

For example, fraud investigators sift through hundreds of fraud alerts daily. Some of them may be real, most are likely not. Eliminating the false positives saves time and money and frees the analyst to focus on important alerts.

 

Charting can be used to summarize the number of alerts by alert type. Modifying the chart to show "total amount of money at risk" reveals that the alert with the most occurrences actually represents only a small portion of the money at risk. By shifting the lens and employing train of thought analysis, analysts can gain valuable insights.

Link analysis can be used to visualize virtually any data set. It is widely being used in social network analysis but is also becoming more common in other applications and industries, such as showing linkages between people-products-stores or suppliers-manufacturers-current orders or patients-drugs-symptoms.

 

Unified data views

In connecting the dots, the key is to be able to quickly and easily explore all of the available, relevant data. Important facts often exist in unrelated systems, making third-party data sources such as social networking sites, blogs, news wires, and network traffic increasingly important.

 

The ability to access these sources and consolidate the data without extensive programming is critical, as the absence of this capability often yields incomplete conclusions. A common complaint, however, is that the analyst needs to employ multiple tools, many of which require the time-consuming construction of complex ETL processes and data warehouses. Because this can be tedious and highly disruptive to a line of reasoning, the ability to easily reach out to these sources from within the analysis framework to create a unified view is vital.

 

One practical example the power of unified data views is checking terrorist watch lists. Consider an analyst who is analyzing airliner passenger manifests and sees the name of someone who was recently reported to the consulate as a potential terrorist.

 

Suspicious, the analyst wants to check a watch list to see if the person is also on that list. She first checks a database of US VISA holders to see if there is a match on name. When she sees he has a current VISA, she checks the watch list and finds the person there as well. Immediately, alarms can go off. The individual in question is about to fly, has a U.S. VISA, and is flagged as someone suspicious on the watch lists. Knowing this, results can be shared using another technique, collaborative analysis.

 

Collaborative analysis

Since investigators and intelligence analysts are often working on interrelated problems, it makes sense that by collaborating, alerting each other to important findings, and making the results available in both a static and live form, individuals can force multiply their efforts across a team and more quickly take action while advancing the analytic process. The ability to document the results of the investigation for audit purposes is also helpful in validating specific findings.

 

Conclusion

Increasing amounts of data coupled with shrinking time windows to understand and act on the information contained within are driving businesses to seek new approaches to analytics. The intelligence community has long been using interactive analytics and link analysis and it is no surprise that the secret is spreading to the commercial BI world.

With interactive analytics the analyst's brain serves as the ultimate pattern recognition machine. The technology allows for unconstrained interactive analysis across disparate data sets. Analysts are empowered to take control of the process and collaborate with others. The result is a more rapid reporting of actionable intelligence.

 

The approach also drastically improves the user experience, which has been too complicated, and is consistent with the way analysts have been trained and think. Most importantly, it allows them to apply their knowledge and experience.

 

Interactive analytics has been put to the test in some of the most demanding applications in the world including cyber crime analysis, counter terrorism, and homeland security applications worldwide. Because of the enormous information challenges organizations face -- and the benefits to be gained -- interactive analytics is taking on new meaning as analysts, given the freedom to explore, leverage technology to solve problems ranging from risk analysis to fraud detection, performance management, customer analysis and more.