As Kx expands into industries beyond financial services, data-driven fields of study, like astronomy, are emerging as use cases for the kdb+ database, with its built-in programming language q.
Kdb+ developers Andrew Magowan and James Neill recently wrote a whitepaper that explores the opportunities for kdb+ in observational astronomy, inspired in part by the construction of new infrastructure that has the potential to record volumes of data that have never before been seen in the field.
The technological advancements that have made new projects like the Large Synoptic Survey Telescope (LSST) in northern Chile and the Square Kilometer Array (SKA) in South Africa possible are causing something of a paradigm shift, and – according to experts – a bit of a headache. Scientists are concerned about their ability to make sense of this newly available data, purely due to its sheer size.
Andrew and James write:
“There is no single choice of programming language in astronomy, but C is used for many astronomical applications. C is one of the most popular and commonly used programming languages in the world with a wide range of uses varying from powering operating systems to building application software. Kdb+ has the ability to extend its functionality through dynamically loaded C/C++ modules, so we have the ability both to make use of existing utilities, and to create our own.
We believe that due to the amount of data collected, its time series nature, and the potential need for both real-time and historical based analysis, kdb+ would be very well suited to the data collected in the astronomy industry, and would be an ideal fit for many future astronomy projects.”
To read the rest of the whitepaper, with code examples, please follow this link (.PDF).
Andrew Magowan and Jame Neill are expert kdb+ programmers and Kx consultants who have worked on large-scale, complex kdb+/q applications at major investment banks around the world. James Neill wrote a previous paper in this series called “An Introduction to Neural Networks with kdb+” available here.