On Time Series Data And A Data Timehouse. Q&A with Ashok Reddy and Darren Coleman  

Ashok Reddy:

Q1. What makes time series data special?

Time series data is special due to its unique properties and the specialized analytical techniques needed to handle it. Key aspects include temporal ordering and dependence, which capture the patterns and dependencies over time. Time series data may exhibit trends and seasonality, which are crucial for understanding underlying processes and making accurate forecasts. Stationarity, autocorrelation, and noise are also essential considerations in time series analysis. The primary goal of time series analysis is forecasting, which requires specialized models and ML techniques as the unique properties of time series data demand tailored approaches, especially now in the context of Generative AI, where the LLM’s don’t handle the time aspects well.

In today’s fast-paced and highly competitive business environment, time series data is becoming increasingly important, with more than 50% of all new data generated is time series or machine data. Understanding the value of time series data and its applications across various industries is crucial for driving growth and staying ahead of the competition. Industries such as finance, manufacturing, telecommunications and healthcare all rely on time series data to uncover trends and correlations that can inform crucial decisions.

KX focuses on helping businesses leverage the power of time series data by bringing analytics and machine learning to the data itself. By eliminating latency and combining various types of data, including time series, relational, and historical data, we enable businesses to generate real-time insights and make faster, better-informed decisions. This paradigm shift from bringing data to analytics to bringing analytics to data is essential for unlocking the full potential of time series data and maximizing customer value.

Q2. Can traditional data warehouses manage time series data?

Traditional data warehouses were designed primarily for storing and managing structured, relational data. While they can technically store time series data, they often struggle to handle the unique challenges and r Scalability: Time series data can grow rapidly due to the continuous collection of data points over time. Traditional data warehouses struggle to handle the large volumes of data efficiently, leading to performance issues and slower query response times.

Data storage: Storing time series data in a traditional data warehouse can be inefficient due to the repetitive storage of timestamps and other time-related attributes. This can result in increased storage requirements and costs.

Query performance: Time series analysis often requires complex queries involving aggregation, filtering, and time-based calculations. Traditional data warehouses may not be optimized for such queries, resulting in slower query performance.

Real-time processing: Time series data often needs to be processed and analyzed in real-time. Traditional data warehouses are typically designed for batch processing, making real-time analysis more challenging.

Specialized functions: Traditional data warehouses may not have built-in support for specialized time series functions, such as window functions, moving averages, or forecasting algorithms, which are crucial for time series analysis.

To overcome these challenges, Data Timehouses are specially designed to handle temporal aspects of data and address these limitations by adding the temporal aspects to data warehouses with the analytical power of time series analysis. These hybrid systems provide a more efficient and scalable solution for managing and processing time-stamped data, enabling businesses to stay agile and competitive in the rapidly evolving digital landscape.

Q3. What is a Data Timehouse?

A data timehouse is a class of data management designed for temporal and vector data. They have three essential elements:

• They’re designed to organize data by time and vector. Although database architects have used traditional database to store data organized by time or vector, those solutions quickly run out of gas. A Data Timehouse is designed from the bottom-up to stored data on disk in temporal order or vector. Optimizing storage format, in turn, speeds up query processing speed for faster answers.
• Accessible By Any Language: A modern Data Timehouse supports data science-centric programming languages like Python, SQL, Q, APIs, Tensorflow, low code and the entire DSML platform / data science toolchain.
• Cloud Native and Integrated with Traditional Data Warehouses on hyperscaler platforms like AWS, Azure, Snowflake, Anaconda

Q4. How do I use a Data Timehouse?

A data timehouse is designed to improve the efficiency and effectiveness of data-driven workflows by blending data science with data through the common denominator of time series. Rather than simply storing time series data in a traditional data warehouse, the data timehouse provides a dedicated platform that enables faster and more cost-effective processing and analysis of time-stamped data.

To use a data timehouse effectively, it’s important to integrate it with existing data warehouse and analytics infrastructures. This allows the data timehouse to handle core workloads, such as querying and analyzing time series data, while still interoperating with other data storage and management systems.

Data timehouses can be particularly valuable for organizations with large volumes of time series data or those in industries where time-sensitive insights are critical for success. By implementing a data timehouse, you can:

• Consolidate your data: Centralize various time-centered datasets, enabling your team to efficiently access and analyze data from multiple sources.
• Optimize data workflows: Streamline the data processing and analysis pipeline, reducing latency and improving overall performance.
• Enhance analytics capabilities: Leverage advanced time series analysis techniques and machine learning models to uncover hidden patterns and insights in your data.
• Improve decision-making: Gain real-time insights and more accurate forecasts, empowering your organization to make better-informed, data-driven decisions.

Ultimately, a data timehouse enables organizations to harness the power of time series data by blending it seamlessly with data science, analytics, and storage capabilities. By implementing this technology, businesses can unlock new opportunities for growth, drive efficiency, and stay ahead in an increasingly competitive and data-driven landscape.

Q5. What does a data timehouse look like?

A data timehouse is a powerful and flexible solution for managing and analyzing time-based data. It centralizes various types of data sets, each centered around time, allowing organizations to efficiently process, analyze, and derive insights from them. The structure and design of a data timehouse may vary depending on the specific requirements and use cases of an organization. However, some common elements and characteristics include:

• Scalable storage: A data timehouse offers scalable storage that can accommodate the growing volume and complexity of time-stamped data generated by digital transformation.
• Advanced analytics engine: Built-in analytics capabilities enable organizations to perform complex time series analysis, anomaly detection, forecasting, and other advanced analytics tasks.
• Interoperability: A data timehouse is designed to work seamlessly with existing data warehouse, analytics, and machine learning platforms, ensuring a smooth integration into an organization’s data infrastructure.
• Data organization: Time-stamped data is organized and indexed based on its temporal characteristics, making it easier for data scientists and analysts to access and analyze relevant data quickly.

In a practical example, a financial services organization using a data timehouse might work with various data sets, such as market data trends, internal order requests, simulation data, and benchmark data. By centralizing these time-centric datasets, the organization can consolidate analytical queries, statistical models, and machine learning applications, leveraging the data timehouse’s unique ability to blend data and analytics, memory and compute, for faster and more cost-effective insights. This enables the organization to make data-driven decisions, optimize strategies, and capitalize on emerging opportunities in the market.

To learn more visit our web site.

Darren Coleman:

Q6. Time series data is critical for modern enterprises. What is special about time series data?

 Temporality is a key attribute of most enterprise datasets. Records associated with timepoints enables corporate organisation to track and analyse changes in critical metrics over time. Common enterprise use cases include changes in market conditions, product sales, changes in socio economic metrics, changing demographics and so on. The common key that unifies such diverse datasets is time. Without the reference of time datasets become static snapshots.

 With more and more electronics devices that are capturing data,  time series has become the most important data of all types.  For example, fitness watches capture HR, Oxygen level, movement (speed, distance etc) and save them as time series. Almost all IOT devices capture video, weather, light and many in time. Hospital medical equipment do continuous capture of vitals. Continuous blood glucose level monitoring is becoming a norm.  Once we have a reasonable amount of historical data (in date and time order). Predictions/Forecasting  and root cause analysis become easy using time series analysis techniques.

In kdb+, data structures are based on ordered lists, unlike classical SQL which is an algebra of sets – inherently unordered (Borror, 2015). Thus, analysis of time-series data using classical SQL can be very computationally intense and overall inefficient. In contrast, by storing data chronologically, temporal operations such as queries, joins and data analysis even at a scale of billions of rows can be instantaneous in kdb+.

References

Borror, Jeffry. Q for Mortals Version 3. 20 Nov. 2015

Q7. Is it possible to manage time series data in traditional data warehouses? If not, why?

Typical relational database is designed to store and manipulate information row-wise. Time series data is inherently ordered by time and most of the queries and analysis is done based on time column.  This requires column based database and specially designed to effectively query, merge data based on time. This is where databases designed specially for time series like kdb+ is critical to manage time series data.

 Column based relational databases can efficiently load only the required columns in memory and manipulated quickly. This makes it much faster in performance compared to row-based traditional data stores.  Kdb+ has withstood the test of time and even after 30 years, remains the system of choice, in Wall Street for critical, real-time analysis of petabyte-scale time series datasets.

 Q8. What is the business of Syneos Health?

Syneos Health® is a leading fully integrated biopharmaceutical solutions organization built to accelerate customer success. The Company translates unique clinical, medical affairs and commercial insights into outcomes to address modern market realities.  

Syneos Health brings together a talented team of professionals, who work across more than 110 countries, with a deep understanding of patient and physician behaviors and market dynamics. Together, colleagues share insights, use the latest technologies and apply advanced business practices to speed Syneos Health customers’ delivery of important therapies to patients.

Q9. You have recently announced a strategic partnership with KX. What is this partnership supposed to deliver?

Together, Syneos Health and KX will deliver data-driven predictive analytics, Artificial Intelligence (AI) and Machine Learning (ML) capabilities. The partnership will help customers address complex healthcare decisions via the industry’s first data timehouse, a new class of data and AI management engine designed for temporal data generated by digital transformation.

Further, through the collaboration, Syneos Health and KX will improve clinical trial efficiency, reduce costs and speed time to market for life-changing therapies for patients. These efforts will empower biopharmaceutical customers to better solve complex healthcare decisions through data – meeting them wherever they are, and supporting their needs, across the clinical to commercial continuum.

Q10. Through the collaboration, Syneos Health and KX aim to improve clinical trial efficiency, reduce costs and speed time to market for life-changing therapies for patients. Can you explain a bit how do you plan to do that?

By working together with KX, Syneos Health is able to dramatically accelerate the most challenging computational challenges with massive data sets, supporting use cases including patient simulations (QSP) for clinical and omnichannel/Real World Evidence for commercial. The acceleration of these computational challenges results in more efficient clinical trials, reduced costs and a decreased time to market for life-changing therapies. 

Q11. Syneos Health also announced a strategic partnership with Microsoft. What role does Microsoft’s Azure technology for the Syneos Health and KX partnership?

Syneos Health’s collaboration with KX amplifies the Company’s partnership with Microsoft as both parties utilize Microsoft’s Azure technology to bring the potential benefits of AI, ML and tech-enabled solutions to customers, providing them with data-driven insights that will enhance performance, effectiveness and efficiency across the asset development lifecycle.

Qx. Anything else you wish to add?

Syneos Health’s technology and data efforts seek to enhance customers’ asset value, with the goal of delivering clinical, medical affairs and commercial solutions informed by industry-leading predictive insights to help unlock efficiencies, accelerate timelines and optimize resource allocation.

To learn more about how Syneos Health is Shortening the distance from lab to life®, visit syneoshealth.com

………………………………………….

Ashok Reddy, CEO, KX

One of the leading voices in vector databases, search & temporal LLM’s, Ashok KX joined as Chief Executive Officer in August 2022. He has more than 20 years of experience leading teams and driving revenue for Fortune 500 and private equity-backed technology companies. He spent ten years at IBM as Group General Manager where he led the end-to-end delivery of enterprise products and platforms for a diverse portfolio of global customers. In addition, he has held leadership roles at CA Technologies and Broadcom, and worked as a special adviser to digital transformation company Digital.AI where he helped the senior leadership team devise the product and platform vision and strategy.

Darren Coleman, Chief Enterprise Data and Intelligence Officer, Syneos Health: