BRIDGING THE BIG DATA CHASM

April 29, 2015 5:07 pm

How engineering and IT can work together to leverage test data
 Competition, market forces, and new innovations require companies to evaluate the people, processes, and technologies used to develop products and services. For test and measurement companies, that evaluation is being driven by the emergence of the Big Analog Data problem, which includes collecting and analysing the raw data from the physical world around us
Unlike the big data typically associated with traditional IT data sources such as social media and enterprise applications, Big Analog Data solutions represent a vastly untapped well of information and insight that test and measurement companies can use to identify and create competitive advantages in data-centric engineering. This is no small feat considering the IDC estimates that only 5 percent of the data collected today is even being analysed.
In this push to better acquire, store, and leverage Big Analog Data solutions, specifically for test data, as it is known in automated test, engineers must start by recognising the role that IT plays in managing it. At present, the sheer amount of data being generated by engineering departments is causing a chasm between IT and engineering. Unless these groups work together to develop tools and methods to better use the data, this chasm will grow deeper.
The first step to cohesion is understanding how big data is classified: structured, unstructured, or semi-structured. Historically, most big data solutions have focused on structured data. Defined by the user, structured data embodies a distinct relationship to the user, who inputs numerous values (name, birthday, address) as raw data. Unstructured data contains no metadata, schema, or other pre-assigned, established organisation.
The third category, semi-structured, is influenced by the dramatic increase in the amount of test data being collected. As more test systems are deployed for 24/7 test data collection, the volume of test data will soon surpass that of human-generated data. Because test data yields so much information, assigning structured value to each and every byte is difficult. Creating hierarchies of data provides structure and makes mining the data after capture easier. This semi-structured test data is typically marked with a timestamp and then analysed across a set time period or for a set stimulus/response event.
Most companies suboptimally implement their solutions for test data because they haven’t anticipated how valuable the correlation of the information gleaned from this stimulus/response data might be at the time of implementation. The most effective methods to combat this issue combine test data analytics with traditional IT tools, but this architecture requires new approaches to data integration and management. This includes new infrastructures and skills to store, mine, and analyse the information-rich data collected. These solutions need to be designed to analyse new data sources and integrate with existing data stores.
Though IT departments haven’t traditionally included test data from engineers and scientists in their overall objectives, they now see the compelling business value to applying analytics and algorithms for exploiting this mountain of data and driving new business opportunities. But making this challenging transition requires asking the following questions:• Is more than half of your analysis manual? • Does your team spend more than five hours a week searching for data trends? • How much data are you actually analysing? Is it less than 80 percent of the data you’re collecting? • Do you have a streamlined process across departments? Or are different teams using different tools? Form a cross-functional data management team To effectively transform into a test data-centric organisation, a cross-functional team should jointly test solutions and ensure compatibility. This team should include a representative from IT, an engineer tasked with data collection, a data scientist, and a manager with a high-level view of how new solutions will roll out to other departments. Additionally, an executive should have a vested interest in the outcome of the inclusion of test data analytics to ensure key members of the cross-functional team are held accountable for progress.

Advertising

Your future advertising space? Our media data