Sensor Data Storage for Industrial IoT
By Michael Donaghy, Global Business Development Executive
Copyright © 2017 Kx Systems, Inc.
Businesses adopting Industrial Internet of Things (IIoT) technologies face a herculean task. In an industry where the number of connected devices and equipment is increasing exponentially, the amount of operational data coming online which must be analyzed and stored rises in tandem. Though many industrial businesses utilizing traditional database management systems find themselves drowning in the sheer volume of sensor data they are producing, those who are forward-looking are turning towards new platforms to help manage their data and, in turn, reduce costs and improve overall performance.
This paper traces the lifecycle of Industrial IoT sensor data from its initial point of entry into a system, through various transformations and tiers of storage, to finally its archival or when disposition.
As a device or sensor generates data, it needs to be ingested and stored where it needs to be analyzed. This is typically close at the physical location of the sensor or device as well as at central facility.
Data first is ingested into an in-memory database where it is processed to perform validations or checks, identify patterns or trends, detect anomalies, and make decisions on this fresh data. As the volumes of data and the richness of processing increase, the amount of available memory can quickly be exhausted.
To optimize the amount of space available in-memory, data can be filtered, that is, it can have any redundancies or noise removed, either at the point of ingestion or just prior to the data being stored to a permanent storage medium.
As data is moved from memory to disk, it can be placed on storage media based on performance and cost criteria, where it can then be compressed to reduce the amount of data which is stored. When the high- precision raw sensor data you have stored has reached the end of its usefulness, you can aggregate or “downsample” that data so that it is reduced in size whilst still maintaining various statistical measures.
These techniques, their implications, and how they function within Kx are explained in greater detail in the sections that follow.