On Predictable Real-Time Data Management Over Persistent Storage Media. Q&A with Andrei Gorine 

Q1. Real-time systems have traditionally relied on in-memory databases to guarantee deterministic performance, while persistence introduces latency unpredictability. How did you architect eXtremeDB/rt 2.0 to bridge this gap? What were the key technical challenges in making persistent storage truly compatible with hard real-time guarantees?

The foundations of a hard real-time in-memory database system are deterministic scheduling, deadline enforcement, rollback behavior, and proving that the system can consistently return control before time runs out. Once those mechanisms are integrated into the database engine, data remains in RAM, where access latency is bounded and execution paths stay visible and measurable. But modern embedded systems increasingly require real persistence, and RAM alone cannot provide that. Persistent storage options such as HDDs, FRAM, MRAM, and other NVRAM technologies all run into practical embedded-system limits involving size, power, capacity, or cost. In practice, NAND flash is the overwhelmingly common option for embedded persistent storage.

NAND flash itself is predictable at the physical device level — read, program, and erase operations have well-defined timing characteristics. The unpredictability comes from the layers above the media. Flash Translation Layers (FTLs) handle mapping, wear leveling, garbage collection, bad-block management, and related I/O functions. In SSDs, eMMC, and SD cards, those mechanisms live inside controller firmware and are opaque to the application and database, effectively turning storage into a black box. Filesystems placed on top add further unpredictability through buffering, journaling, write reordering, and deferred allocation.

The opportunity to intervene exists in unmanaged (raw) flash configurations, where eXtremeDB/rt 2.0 integrates a transactional flash translation layer (TFTL) directly into the database kernel, replacing conventional FTL functionality with a deterministic, transaction-aware storage layer. The engine directly manages mapping, allocation, wear leveling, recovery, and flash operation ordering. Because the kernel directly tracks copy-on-write activity and recovery state, it eliminates redundant translation layers, reduces timing variability and write amplification, and keeps persistent operations compatible with deadline-aware scheduling.

Another major challenge was continuous timing verification. eXtremeDB/rt places high-resolution timing checkpoints around flash I/O, page management, and lock waits so the database kernel can determine during execution whether a transaction can still meet its deadline. Because modified pages are explicitly tracked throughout the entire stack — from upper-layer copy-on-write processing down to media-driver operations, rollback cost becomes tied to a known write set rather than uncontrolled storage activity, making rollback timing analyzable even in the presence of persistence.

Q2. Your Transactional Flash Translation Layer (TFTL) claims to eliminate the unpredictable latency inherent in managed flash. Can you explain what makes standard flash controllers found in managed flash unsuitable for hard real-time systems? How does TFTL’s deadline-aware approach fundamentally differ from a controller’s typical wear-leveling and garbage collection strategies?

Flash controllers used in SSDs, eMMC, SD cards, and similar managed flash devices are generally difficult to use in hard real-time systems because their internal FTLs are optimized mostly for flash longevity, reliability, and overall storage performance rather than predictable timing behavior. Controller firmware can start internal flash-management operations while a real-time transaction is already executing. This usually happens reactively — for example when free space becomes tight or wear leveling needs adjustment. The resulting latency spikes are difficult to predict analytically and happen outside application and database control.

In raw flash configurations, eXtremeDB/rt 2.0 integrates flash management and persistence directly into the database kernel through its Transactional Flash Translation Layer (TFTL). TFTL still performs the normal responsibilities of an FTL, including wear leveling, flash management, and reliability handling. The difference is that these operations are coordinated with transaction scheduling and timing verification instead of being triggered independently whenever the flash-management logic decides they are needed, without coordination with hard real-time scheduling constraints. In other words, TFTL still manages the flash, but it does so with hard real-time timing constraints in mind. That keeps persistence activity visible to the kernel and allows it to be accounted for before it affects execution timing. 

Q3. eXtremeDB/rt 2.0 offers multiple configurable deadline policies—from strict enforcement to dynamic adaptation to application-controlled handling. What real-world scenarios drove the need for this flexibility? How should developers think about choosing the right deadline policy for their specific use case, and what are the consequences of getting it wrong?

First, it is important to understand that in many real-time systems, controlling time is more important than collecting every piece of data or making the most “informed” decision possible. For example, in avionics, using terrain or sensor data that is no longer current may be far more dangerous than skipping that data entirely and continuing with a timely control decision. This is one of the fundamental reasons real-time databases exist in the first place: they sit directly in the path of collecting and processing time-sensitive external inputs.

Different real-time applications place different timing constraints on database operations, which is why eXtremeDB/rt 2.0 provides two preconfigured deadline policies: Strict and Adaptable. Deadlines themselves are always enforced by the database kernel. The difference lies in how tightly the application schedule is coupled to database timing versus database consistency once a deadline is approached. In some systems, allowing transactions to execute as long as possible within the available schedule is more important than immediately preserving logical consistency. For example, in an industrial controller, updating a single sensor value may be secondary to issuing a control decision within a fixed time window. In other systems — such as sensor-fusion or telemetry-correlation applications — the control decision itself depends on having a logically consistent set of correlated sensor readings, so transactions may need to be interrupted earlier to guarantee rollback and preserve database consistency before the deadline expires.

The Strict deadline policy favors temporal database consistency over logical consistency. A transaction is allowed to run until its exact deadline, after which database execution is interrupted immediately. The transaction may remain logically incomplete, and initiating rollback to restore database consistency (which is still possible, of course), becomes the application’s responsibility. This approach fits systems such as industrial control, avionics, or protection logic, where returning control on time matters more than preserving intermediate database state.

The Adaptable policy makes the opposite tradeoff. During execution, the kernel continuously estimates rollback cost and interrupts the transaction and initiates the rollback early enough to guarantee that rollback can complete before the deadline. That keeps the database logically consistent while still operating within real-time constraints. This model is often a better fit for telemetry, monitoring, or sensor aggregation systems where occasional timing adaptation is acceptable.

Choosing the wrong policy usually means optimizing for the wrong kind of failure. Overly strict enforcement can lead to excessive transaction interruption and reduced throughput even when the application could tolerate earlier interruption to preserve logical consistency. On the other hand, policies that reserve rollback time too aggressively may interrupt transactions unnecessarily early and reduce useful work. In practice, the choice comes down to whether the system prioritizes preserving temporal consistency alone or preserving both temporal and logical consistency together. 

Q4. You’ve dramatically reduced the footprint while adding persistence, flash management, and a RESTful server. What architectural decisions or trade-offs enabled this? As embedded systems become more capable yet remain resource-constrained, how do you balance feature richness against the minimalism that embedded developers demand?

First, “footprint” in embedded systems needs to be understood carefully. Application code can easily exceed available RAM because it often does not reside in RAM at all. Many embedded processors execute firmware directly from flash using XIP (Execute In Place), while RAM is used mainly for runtime data: stack, variables, buffers, caches, metadata, and application state. So firmware image size and actual runtime RAM requirements are often two very different things.

For eXtremeDB/rt, the more important constraints are runtime RAM footprint and stack usage. Executable size is still relevant, but in many embedded systems the tighter constraints are runtime RAM, cache requirements, stack usage, and timing behavior rather than total image size. Even modern STM32 and NXP MCU-class systems often operate with only a few hundred kilobytes to a few megabytes of RAM once RTOS services, networking stacks, drivers, caches, and application logic are taken into account.

Furthermore, adding flash management was not really optional. Most embedded systems use flash in one form or another, and many rely on it as their primary persistent storage medium. Once the database itself is expected to provide persistence on top of flash storage, flash handling effectively becomes part of the database problem itself rather than something delegated entirely to external storage layers (SSD controller, FTL, file systems, etc). That naturally pushed the architecture toward integrating flash management directly into the database kernel instead of layering it as a large independent subsystem. Once flash management was integrated into the kernel and the core real-time mechanisms were in place, the focus shifted toward reducing runtime cost as much as possible. The first step was designing eXtremeDB/rt to be highly configurable at the source-code level, allowing functional components to be compiled in or out depending on application requirements. For example, only the necessary data types or index structures may be included in a particular kernel build. The kernel itself is then optimized by minimizing internal buffers, moving allocation maps and bitmaps from RAM to persistent media where practical, and using more compact internal structures. For example, instead of maintaining a full bitmap in memory, a ring buffer stored on persistent media may be used. Techniques such as lossy bitmaps can further reduce metadata overhead.

These decisions are not without tradeoffs. Ring buffers may slightly increase persistent storage size, while access latency can increase more significantly, although still predictably. Lossy bitmaps may require additional processing to determine which pages have actually changed. The overall impact depends heavily on database access patterns and workload characteristics, which is why applications can choose whether to enable these and other memory-optimization techniques depending on available RAM, latency requirements, and access patterns.

A similar architectural approach was taken with remote-management support and REST services. Many embedded devices are traditionally deployed as standalone systems, but increasingly include networking through built-in Ethernet MAC controllers. Industrial monitoring sensors are a common example: devices may be located deep inside production lines where manual access is difficult or expensive. As a result, connectivity is often needed for remote monitoring, configuration, and software deployment. To support these use cases, we implemented a lightweight integrated RESTful server with TLS support. The REST server core itself is under 10 KB, with additional REST-based services included for monitoring and configuration workflows. These services are, of course, outside the core real-time processing path, but they become indispensable both as development tools and for monitoring deployed systems.

Q5. While hard real-time databases have traditional homes in avionics and industrial automation, what emerging application domains are driving demand for deterministic data management? Are there new classes of problems—perhaps in autonomous systems, 5G/6G infrastructure, or edge AI—where hard real-time databases are becoming mission-critical?

In the database industry, “real-time” is often associated with performance or low latency, but hard real-time systems are really about deterministic timing. Operations must complete within defined deadlines, and once deadlines are missed the system can no longer guarantee correct behavior even if the computed result itself is accurate. Traditionally, real-time systems were mostly tight control loops reading sensors and driving actuators on fixed schedules. Databases, if present at all were usually secondary components used for logging, diagnostics, or post-event analysis. That model is changing. Modern embedded systems continuously process and act on live data in real time, making data management part of the critical execution path rather than a background subsystem. Once the data layer participates directly in runtime decision-making, bounded and predictable database behavior becomes just as important as deterministic task scheduling. Even perfectly deterministic control logic cannot compensate for delayed or inconsistent data delivery.

The clearest growth area is autonomous and semi-autonomous embedded systems. While aerospace, industrial automation, and military systems remain major application areas for hard real-time technology, similar requirements are now appearing in autonomous robotics, advanced medical devices, and other systems expected to operate with limited human intervention. We have seen this ourselves in projects such as laboratory analyzers coordinating reagent dosing, sample movement, and sensor acquisition under strict schedules, or infusion pumps continuously adjusting medication flow based on changing patient vital statistics. These systems combine sensor data and internal state while making decisions under bounded timing constraints, where stale or partially processed data can be worse than missing data entirely. In these environments, timing itself becomes part of the system’s correctness guarantees.

Edge AI and 5G/6G infrastructure are somewhat different because they are not usually hard real-time in the classical sense. These are what would normally be classified as “soft real-time” systems, meaning that staying within timing targets and maintaining predictable low latency is highly desirable, but deadline misses are tolerated. Missing a deadline there typically degrades throughput, responsiveness, or quality of service rather than causing immediate or catastrophic failure. At the same time, unlike many traditional hard real-time systems built around relatively small microcontrollers and tightly bounded control loops, these platforms often operate on substantially larger hardware with accelerators, larger RAM footprints, managed flash storage, and significantly larger persistent datasets. That naturally pushes them toward more deterministic handling of large amounts of local state and persistent data.

Qx. Anything else you wish to add?

One thing I would add is that the embedded database industry probably needs to pay more attention to systems built around managed flash storage. These systems are not “hard real-time” in the strict sense because managed flash devices — SSDs, eMMC, UFS, and similar storage introduce internal behavior that higher-level software, including database systems, cannot fully control or predict. We discussed Edge AI and 5G/6G edge systems earlier. These systems often keep machine-learning models, embeddings, logs, configuration data, software containers, and updates directly on the device. Managed flash is commonly used because it provides large storage capacity and is easy to integrate.

This is one of the reasons eXtremeDB/rt introduced xFile (Transactional File Mapping). xFile allows eXtremeDB/rt to store databases either on top of an operating-system file system or directly on managed block storage exposed by the device’s FTL. xFile couples the database transaction mechanism directly with media access, significantly reducing I/O overhead and avoiding multiple buffering and filesystem layers that otherwise increase RAM usage and latency variability. It uses a page-mapped copy-on-write recovery mechanism for reliable transaction recovery on managed flash devices.

…………………………………….

Andrei Gorine, Chief Technology Officer at McObject

McObject co-founder Andrei Gorine leads the company’s product engineering. As CTO, he has driven the growth of the eXtremeDB embedded database system, from the product’s conception to its current wide usage in virtually all embedded systems market segments. Mr. Gorine’s strong background includes senior positions with leading embedded systems and database software companies; his experience in providing embedded storage solutions in such fields as industrial control, industrial preventative maintenance, satellite and cable television, and telecommunications equipment is highly recognized in the industry. Mr. Gorine has published articles and spoken at many conferences on topics including real-time database systems, high availability, and memory management. Over the course of his career he has participated in both academic and industry research projects in the area of real-time database systems. Mr. Gorine holds a Master’s degree in Computer Science from the Moscow Institute of Electronics and Mathematics, and is a member of IEEE and ACM.