{"id":5408,"date":"2021-08-13T09:05:53","date_gmt":"2021-08-13T09:05:53","guid":{"rendered":"http:\/\/www.odbms.org\/blog\/?p=5408"},"modified":"2021-08-13T19:35:58","modified_gmt":"2021-08-13T19:35:58","slug":"on-time-series-databases-interview-with-ryan-betts","status":"publish","type":"post","link":"https:\/\/www.odbms.org\/blog\/2021\/08\/on-time-series-databases-interview-with-ryan-betts\/","title":{"rendered":"On Time Series Databases. Interview with Ryan Betts"},"content":{"rendered":"\n

I have interviewed Ryan Betts<\/strong>, <\/em>VP of Engineering at InfluxData<\/a>. We talked about time series databases, InfluxDB and the InfluxData stack. RVZ<\/p>\n\n\n\n

“Time series databases have key architectural design properties that make them very different from other databases. These include time-stamped data storage and compression, data lifecycle management, data summarization, ability to handle large time-series-dependent scans of many records, and time-series-aware queries.<\/em>“–Ryan Betts<\/strong><\/p><\/blockquote>\n\n\n\n

Q1. What is time series data?<\/h3>\n\n\n\n

Ryan Betts<\/strong>: Time series<\/a> data consists of measurements or events that are captured and analyzed, often in real time, to operate a service within an SLO<\/a>, detect anomalies, or visualize changes and trends. Common time series applications include server metrics, application performance monitoring, network monitoring, and sensor data analytics and control loops. Metrics, events, traces and logs are examples of time series data.<\/p>\n\n\n\n

Q2. What are the hard database requirements for time series applications?<\/h3>\n\n\n\n

Ryan Betts<\/strong>: Managing time series data requires high-performance ingest (time series data is often high-velocity, high-volume), real-time analytics for alerting and alarming, and the ability to perform historical analytics against the data that\u2019s been collected. Additionally, many time series applications apply a lifecycle policy to the data collected \u2014 perhaps downsampling or aggregating raw data for historical use.  <\/p>\n\n\n\n

With time series, it\u2019s common to perform analytics queries over a substantial amount of data. Time series queries commonly include columnar scans, grouped and windowed aggregates, and lag calculations. This kind of workload is difficult to optimize in a distributed key value store. InfluxDB<\/a> uses columnar database techniques to optimize for exactly these use cases, giving sub-second query times over swathes of data and supporting a rich analytics vocabulary.<\/p>\n\n\n\n

While time series data is typically structured, it often has dynamic properties that aren\u2019t well-suited to strict schema enforcement. Time series databases often specify the structure of data but allow schema-on-write. Another way of saying this is that time series databases often support arbitrary dimension data to decorate the contents of the fact table. This allows developers to create new instrumentation or collect metrics from new sources without performing frequent schema migrations. Document databases and column-family stores similarly allow flexible schema in their own contexts. The motivation with time series is similar \u2014 optimizing for developer productivity.<\/p>\n\n\n\n

In addition to high-performance ingest, non-trivial analytics queries, and flexible schema, TSDBs also need to bridge real-time analytics to real-time action. There\u2019s little point doing real-time monitoring if you can\u2019t also automate real-time responses. So time series databases, like other real-time analytics systems, need to provide the analytics function and the ability to tie into real-time operations. That means integrating automated alerting, alarming, and API invocations with the query analytics performed for monitoring. <\/p>\n\n\n\n

Q3. How do you manage the massive volumes and countless sources of time-stamped data produced by sensors, applications and infrastructures?<\/h3>\n\n\n\n

Ryan Betts<\/strong>: The InfluxData stack<\/a> is optimized for both regular (metrics often gathered from software or hardware sensors) and irregular time series data (events driven either by users or external events), which is a significant differentiator from other solutions like Graphite<\/a>, RRD<\/a>, OpenTSDB<\/a>, or Prometheus<\/a>. Many services and time series databases support only the regular time series metrics use case. <\/p>\n\n\n\n

InfluxDB lets users collect from multiple and diverse sources, store, query, process and visualize raw high-precision data in addition to the aggregated and downsampled data. This makes InfluxDB a viable choice for applications in science and sensors that require storing raw data.<\/p>\n\n\n\n

At the storage level, InfluxDB organizes data into a columnar format and applies various compression algorithms, typically reducing storage to a fraction of the raw uncompressed size. Time series applications are \u201cappend-mostly\u201d.  The majority of arriving data is appended.  Late arriving data and deletes occur with some frequency \u2014 but primarily writes result in appending to the fact table. The database uses a log structured merge tree architecture to meet these requirements. Deletes are recorded first as tombstones and are later removed through LSM compaction.<\/p>\n\n\n\n

Q4. Can you give us some time series examples?<\/h3>\n\n\n\n

Ryan Betts<\/strong>: Time series data, also referred to as time-stamped data, is a sequence of data points indexed in time order. Time-stamped is data collected at different points in time.<\/p>\n\n\n\n

These data points typically consist of successive measurements made from the same source over a time interval and are used to track change over time.<\/p>\n\n\n\n

Weather records, step trackers, heart rate monitors, all are time series data. If you look at the stock exchange, a time series tracks the movement of data points, such as a security\u2019s price over a specified period of time with data points recorded at regular intervals.<\/p>\n\n\n\n

InfluxDB has a line protocol for sending time series data which takes the following form:<\/p>\n\n\n\n

<measurement name>,<tag set> <field set> <timestamp><\/p>\n\n\n\n

The measurement name is a string, the tag set is a collection of key\/value pairs where all values are strings, and the field set is a collection of key\/value pairs where the values can be int64, float64, bool, or string. The measurement name and tag sets are kept in an inverted index which makes lookups for specific series very fast.<\/p>\n\n\n\n

For example, if we have CPU metrics:<\/p>\n\n\n\n

cpu,host=serverA,region=uswest idle=23,user=42,system=12 1549063516<\/p>\n\n\n\n

Timestamps<\/a> in InfluxDB can be by second, millisecond, microsecond, or nanosecond precision. The micro and nanosecond scales make InfluxDB a good choice for use cases in finance and scientific computing where other solutions would be excluded. Compression is variable depending on the level of precision the user needs.<\/p>\n\n\n\n

Q5. The fact that time series data is ordered makes it unique in the data space because it often displays serial dependence. What does it mean in practice?<\/h3>\n\n\n\n

Ryan Betts<\/strong>: Serial dependence occurs when the value of a datapoint at one time is statistically dependent on another datapoint at another time.<\/p>\n\n\n\n

Though there are no events that exist outside of time, there are events where time isn\u2019t relevant. Time series data isn\u2019t simply about things that happen in chronological order \u2014 it\u2019s about events whose value increases when you add time as an axis. Time series data sometimes exists at high levels of granularity, as frequently as microseconds or even nanoseconds. With time series data, change over time is everything.<\/p>\n\n\n\n

Q6. How is time series data understood and used?<\/h3>\n\n\n\n

Ryan Betts<\/strong>: Time series data is gathered, stored, visualized and analyzed for various purposes across various domains:<\/p>\n\n\n\n

  1. In data mining<\/a>, pattern recognition<\/a> and machine learning<\/a>, time series analysis is used for clustering, classification, query by content, anomaly detection and forecasting.<\/li>
  2. In signal processing<\/a>, control engineering<\/a> and communication engineering<\/a>, time series data is used for signal detection and estimation.<\/li>
  3. In statistics, econometrics, quantitative finance<\/a>, seismology<\/a>, meteorology<\/a>, and geophysics<\/a>, time series analysis is used for forecasting.<\/li><\/ol>\n\n\n\n

    Time series data can be visualized in different types of charts to facilitate insight extraction, trend analysis, and anomaly detection. Time series data is used in time series analysis (historical or real-time) and time series forecasting to detect and predict patterns \u2014 essentially looking at change over time. <\/p>\n\n\n\n

    Q7. You also handle two other kinds of data, namely cross-section and panel data. What are these? How do you handle them?<\/h3>\n\n\n\n

    Cross-sectional data<\/a> is a collection of observations (behavior) for multiple entities at a single point in time. For example: Max Temperature, Humidity and Wind (all three behaviors) in New York City, SFO, Boston, Chicago (multiple entities) on 1\/1\/2015 (single instance).<\/p>\n\n\n\n

    Panel data<\/a> is usually called cross-sectional time series data, as it is a combination of both time series data and cross-sectional data (i.e., collection of observations for multiple subjects at multiple instances).<\/p>\n\n\n\n

    This collection of data can be combined in a single series, or you can use Flux lang<\/a> to combine and review this data to gather insights. <\/p>\n\n\n\n

    Q8. There are several time series databases available in the market. What makes InfluxDB time series database unique?<\/h3>\n\n\n\n

    Ryan Betts<\/strong>: When doing a comparison, the entire InfluxDB Platform should be taken into account. There are multiple types of databases that get brought up for comparison. Mostly, these are distributed databases like Cassandra<\/a> or more time-series-focused databases like Graphite<\/a> or RRD<\/a>. When comparing InfluxDB with Cassandra or HBase<\/a>, there are some stark differences. First, those databases require a significant investment in developer time and code to recreate the functionality provided out of the box by InfluxDB. Finally, they\u2019ll have to create an API to write and query their new service.<\/p>\n\n\n\n

    Developers using Cassandra or HBase need to write tools for data collection, introduce a real-time processing system and write code for monitoring and alerting. Finally, they\u2019ll need to write a visualization engine to display the time series data to the user. While some of these tasks are handled with other time series databases, there are a few key differences between the other solutions and InfluxDB. First, other time series solutions like Graphite or OpenTSD<\/a>B are designed with only regular time series data in mind and don\u2019t have the ability to store raw high-precision data and downsample it on the fly.<\/p>\n\n\n\n

    While with other time series databases, the developer must summarize their data before they put it into the database, InfluxDB lets the developer seamlessly transition from raw time series data into summarizations.<\/p>\n\n\n\n

    InfluxDB also has key advantages for developers over Amazon Timestream<\/a>. Among them:<\/p>\n\n\n\n