On Shakti’s data platform and the STAC-M3™ Benchmark Council for Financial Trading Applications. Q&A with Fintan Quill
Q1. You have recently announced “Record Breaking Results” on the STAC-M3™ Benchmark for Financial Trading Applications. What is the STAC-M3™ Benchmark for Financial Trading Applications?
STAC is the Securities Technology Analysis Center, an organization setup to bring financial firms and technology vendors together and provide industry benchmarks based on real-world trading applications.
STAC-M3 is one of those benchmarks. It measures the performance of high-speed analytics on time-series data, such as tick-by-tick market data. For further information readers can check out the following link: https://stacresearch.com/m3
Q2. How does the process work for a company to have authorized public release of audited STAC-M3 Benchmark results?
Leading financial firms and vendors in the STAC Benchmark Council developed STAC-M3, which ensures that benchmarks focus on real business needs and represent the workloads and operations that user firms deal with daily.
For a database company to participate in STAC-M3, it first needs to develop a STAC Pack, which is software that enables a given STAC Test Harness to conduct tests of a specific product.
Then a vendor (in this case Shakti and our storage partner DDN) put together a complete hardware and software stack to be tested (the stack under test, or SUT) and engage STAC to perform an independent, third-party audit. STAC’s independence, along with the resulting STAC Reports, Configuration Disclosures, and other documentation, ensures that the audit is repeatable, fair, and comparable to other tested systems.
Q3. Financial firms on the STAC Benchmark Council designed STAC-M3 to test a representative range of business use cases. What are such use cases?
There are a variety of tests that compose a mixture of compute intensive, storage intensive, and cache-based calculations that can be serial or highly parallel in nature.
There are common aggregate queries such as Volume Weighted Average Price (VWAP), National Best Bid Offer (NBBO) as well as bi-temporal and point-in-time queries such as Market Snap.
The combination of these tests creates a rigorous benchmark with which to evaluate the performance of a tech stack without having to go through the sometimes onerous process of an in-house evaluation, thereby saving time and money for member trading firms of STAC.
Q4. Why do such use cases need fast and efficient time-series analysis?
With the birth of electronic trading decades ago the financial trading industry has long had a need for capturing time-series data in real-time to help make trading decisions. It also requires the storage and analysis of this data for regulatory and risk reporting, as well as for backtesting/simulation of trading strategies before deploying them into the real world.
The financial services industry has been ahead of other industries in its use of time-series data, however other industries such as Internet of Things (IoT) and motorsport telemetry data are now adopting these types of data sets, which Shakti supports natively.
Q5. Tell us a bit about Shakti’s data platform. What are the key benefits?
Shakti’s core principles are performance and efficiency. In the financial services industry, time is of the essence with trading decisions made at nanosecond levels, therefore performance is key.
Shakti is incredibly efficient in terms of its hardware footprint, in particular for data storage both in-memory and on-disk, as can be seen in the STAC-M3 record-breaking results. This helps reduce the number of machines required to run an application, which represents a great cost saving as well as lowering the energy consumption required for computing, a topic which is top of mind for many firms these days.
Time-series support is native to the platform, allowing for analysis and joining of data sets at different granularities. There is also support for C, Python, NodeJS and Java, as well as an in-built foreign function interface (FFI) allowing programmers to re-use existing well-established libraries written in other languages without having to reinvent the wheel.
Q6. How is Shakti’s data platform used by DDN®?
Speed needs speed. DDN is the world’s premier high-performance storage provider. Their storage appliances can be accessed by Shakti from a single node or scaled across a multi-node setup.
This scalability can be incredibly useful across trading firms who wish to increase the number of disparate trading teams accessing the same common datasets, such as market data, while maintaining optimal performance.
Q7 What are the main results you have obtained for the STAC-M3™ Benchmark for Financial Trading Applications?
Shakti was able to set a new record for storage efficiency and attain ‘state of the art’ benchmark performance with only one server. The combination of the best storage efficiency result alongside a small compute workload footprint, enables our customers to get the high performance time-series analytics they demand, using their current industry standard equipment.
More information about our benchmark is available in the STAC announcement: https://www.stacresearch.com/news/SHK211203
Q8. Are these results specific to a class of business use cases? Which ones?
There are a variety of use-cases where these results can be applied. Among these would be pre- and post-trade risk checks/reporting, regulatory reporting, historical research, backtesting, real-time trading/signal generation as well as standard OLAP/OLTP workloads.
The time-series support within Shakti allows for temporal aggregation, bi-temporal analysis (i.e. merging multiple time-series) and further fine-grained analysis due to the support of multiple time types, both cardinal and ordinal, including microsecond and nanosecond support.
………………………………
Fintan Quill heads Sales Engineering at Shakti, which he joined in 2019. An expert in complex database analytics, he has worked over the past 16 years at a number of Wall Street investment banks, hedge funds, and trading shops, building critical infrastructure for trading analytics and other purposes. Fintan is a graduate of Trinity College, Dublin with a specialization in Computing and Microelectronic Engineering. He is a frequent speaker on database topics, including at Seattle Data Day, Austin Data Day, Strata+Hadoop World, the Irish Network USA and various Big Data Meetups around North America and Europe.