￼Pipelining Vector-Based Statistical Functions for In-Memory Analytics
A New Technique to Reduce Latency in Managing Market Data
Copyright 2013, McObject LLC
Managing trade- and quote-related market data is a key task for software underlying today’s automated capital markets, including applications for algorithmic trading, risk management and order matching and execution. Reducing latency in these systems can deliver a competitive advantage, so technology that accelerates market data management is eagerly welcomed.
Market data typically takes the form of a time series, or repeated measurements of some value over time. Database management systems (DBMSs) have evolved specialized techniques to speed up time series processing, including column-based handling of such data. This paper examines the column-based approach as implemented by McObject’s eXtremeDB Financial Edition database system. It looks at how eXtremeDB Financial Edition implements columns, with database designs that support hybrid column- and row-based data handling – and focuses on a key, related eXtremeDB Financial Edition feature for reducing latency: its library of vector-based statistical functions, which are designed to execute over data sequences (columns). The paper shows how columnar storage improves performance by maximizing the proportion of relevant market data that is loaded into CPU cache, and how pipelining of vector-based statistical functions maximizes performance by eliminating costly transfers between CPU cache and DRAM.
DOWNLOAD REPORT (.PDF): pipelining-white-paper