The Latency Dilemma: Steve Graves on Why Standard Flash Management Fails Time-Critical Systems

Q1 Your white paper (*) makes a compelling case that traditional Flash Translation Layers (FTLs) – whether vendor-provided or open-source – fundamentally conflict with real-time database requirements. Can you explain the specific mechanisms by which conventional FTLs introduce non-deterministic latency, and why the industry-standard approach of abstracting NAND flash management away from the application layer is particularly problematic for time-critical systems?

More specifically: What happens when a traditional FTL’s garbage collection or wear-leveling operation collides with a database transaction that has a hard deadline? How do managed NAND solutions (eMMC, UFS, NVMe SSDs) make this problem even worse, and under what circumstances might developers actually prefer the complexity of raw NAND despite the additional development burden?

The premise that NAND flash is inherently non-deterministic is actually a common misconception. At the physical hardware level, individual raw NAND operations—reads, programs, and erases—have fixed, well-defined execution times dictated by electrical characteristics. The unpredictability arises entirely from the Flash Translation Layer (FTL) and NAND’s erase-before-write constraint, which forces out-of-place updates. Standard FTLs abstract this complexity behind a generic block device interface, but to maximize throughput and endurance, they execute background tasks like garbage collection, wear leveling, and bad block remapping asynchronously. As a result, a tiny, time-critical write can suddenly freeze while the FTL reads valid pages, rewrites them elsewhere, and erases a block. 

Because standard FTLs prioritize average-case throughput over worst-case bounds, latency becomes state-dependent—governed by free space, fragmentation, and recent workload history. Managed NAND devices (eMMC, UFS, NVMe) compound this opacity by burying aggressive firmware heuristics, multi-stream GC, and deep queues behind fixed host interfaces, offering zero visibility or control over when background work occurs. A real-time database can meticulously schedule its own execution, only to be completely derailed by multi-millisecond storage stalls. 

This is why abstracting flash management away behind a conventional block device is fundamentally problematic for time-critical systems. The goal isn’t to force application developers to write low-level hardware drivers, but to move flash management into the real-time database kernel via a Transactional FTL (TFTL). By merging copy-on-write transaction mapping with FTL page management and garbage collection, the database regains control over storage operations, allowing it to execute flash maintenance deterministically while guaranteeing hard transaction deadlines. 

Q2. The concept of merging Copy-on-Write (CoW) database transaction management with Flash Translation Layer address mapping is intellectually elegant, but the devil is in the details. Can you walk us through the core architectural decisions behind TFTL’s design?

Specifically:

• How does TFTL’s commit group structure (with control pages serving as both transaction boundaries and address mapping indices) differ from traditional approaches that keep transaction logs separate from storage management?
• Your paper mentions that lookup operations require at most log₂(max Laddr) steps – for a database with a million logical pages, that’s 20 transitions. How does the control block cache optimization work in practice, and what hit rates are you seeing?
• Most importantly: What tradeoffs did you make in TFTL’s design? Are there workload patterns where a traditional FTL + transaction logging approach might actually outperform TFTL, or specific flash characteristics (SLC vs. MLC vs. TLC) that change the optimization landscape?

1. Unified Architecture vs. Dual-Layer Logging

Traditional storage stacks operate two disconnected engines: a database write-ahead log (WAL) tracking logical transactions, and an FTL managing physical translation, wear leveling, and garbage collection. Because these layers communicate across an opaque block device boundary via flush barriers, the database remains completely blind to physical storage state. TFTL replaces this fragmented approach by merging transaction boundaries directly into the storage mapping using commit groups. 

 As data pages are written sequentially, a commit group closes with a control page that acts as both the atomic commit record and the logical-to-physical address index. Control pages link back to form a versioned mapping chain, eliminating the need for a separate database log or FTL metadata state. Physical append order directly mirrors logical commit order, turning crash recovery into a straightforward, deterministic scan of committed control pages without any inter-layer reconciliation. 

2. Address Lookup & Control Block Caching

Un-cached lookups traverse control page pointers backward, which establishes a strict worst-case bound of log₂(max Laddr) transitions—or roughly 20 steps for a million logical pages. In practice, TFTL avoids chasing control pages across flash media by utilizing an in-memory control block cache structured as a hash table with per-bucket LRU eviction. 

Because real-time database workloads display heavy spatial and temporal locality, active page mappings naturally cluster inside recent control blocks that stay pinned in RAM. “The actual cache hit rate depends on the cache size and the database access pattern, but in practice we typically see around 90%. The logarithmic bound functions as a provable fallback for cold reads rather than the typical execution path. 

3. Design Tradeoffs & Media Dynamics

TFTL explicitly trades aggressive average-throughput optimizations for latency determinism. Can transaction logging outperform TFTL? In principle, yes, you can probably come up with scenarios where that happens. For example, with very small write transactions, TFTL may still have to fill an entire control group, which is not ideal. On the other hand, how efficient a log-based approach would be in that same situation really depends on the third-party FTL underneath. Sometimes it helps, sometimes it just hides the same costs in a different place. It’s not that straightforward.

Flash technology (SLC/MLC/TLC) mostly affects cost and the number of erase cycles. Usually the cheaper the flash, the fewer cycles you get. Other parameters differ as well, but in practice we don’t really treat them differently.

It bears repeating, that the metric of success for a real-time database is different than for an OLTP database. For OLTP, the metric is throughput, or maximum transactions-per-second. For a real-time database, the metric is in-time transactions. Speed is secondary to meeting deadlines.

Q3. The conventional wisdom is simple: NOR flash for code execution, NAND flash for data storage. But your paper hints at more nuance, particularly the statement that “NOR flash’s byte-level read access makes it more compatible with in-memory database access methods rather than traditional disk-based block I/O.”

 Can you expand on this? For embedded systems architects designing real-time databases from scratch, under what conditions would a hybrid approach (NOR for critical metadata/indices, NAND for bulk data) make sense? And more provocatively: As NOR flash technology evolves and costs potentially decrease, could we see a resurgence of NOR-based persistent databases for specific real-time applications where the “erase-before-write” constraint of NAND is simply too expensive from a latency perspective?

NOR flash’s byte-level random-read addressability allows memory cells to be mapped directly into the CPU address space, effectively functioning like read-only RAM. Unlike NAND, which forces page-based block I/O and buffer management, NOR lets a database runtime chase pointers directly through trees and index structures without page-fetching overhead, FTL translation, or garbage collection stalls. This aligns naturally with in-memory database access patterns, where operations consist of fine-grained, read-heavy traversals rather than coarse block reads. However, because NOR still suffers from slow write speeds, lower density, and higher costs, its utility remains focused on read-dominant scenarios. 

A hybrid architecture makes sense when timing constraints differ across data types. In this model, NOR hosts small, latency-critical state—such as calibration parameters, safety thresholds, or index roots—where a single storage stall would violate real-time deadlines. Meanwhile, NAND handles bulk data, time-series telemetry, and event histories where high storage density and sequential throughput outweigh single-byte lookup latency. 

Regarding a resurgence of NOR-based persistent databases: while declining costs could make NOR attractive for niche, read-mostly embedded systems requiring zero-latency startup, physical constraints remain a barrier. NOR still requires block-level erases before writes and offers significantly lower write speeds and storage density than NAND. For systems handling dynamic data, solving the erase-before-write problem on NAND through an integrated, transactional FTL remains a far more scalable and cost-effective path to real-time determinism. 

Q4. Garbage collection is traditionally the enemy of determinism, whether we’re talking about programming language runtimes or flash memory management. Yet TFTL implements garbage collection that operates within real-time constraints. How?

Can you detail:

• The specific conditions under which TFTL’s garbage collector is triggered, and how its execution time is bounded?
• How does the tail-to-head relocation strategy (copying reachable pages from tail to head) interact with wear-leveling requirements? Doesn’t moving “hot” pages that are frequently updated create wear concentration near the head pointer?
• Your benchmark showed 4x reduction in I/O operations with xflash vs. xfile – how much of this improvement comes from eliminating redundant mapping layers vs. smarter garbage collection strategies?

• For developers working in safety-critical domains (aerospace, medical devices, automotive), what formal methods or worst-case execution time (WCET) analysis techniques can be applied to TFTL’s garbage collection to certify its real-time behavior?

TFTL abandons the traditional model of background garbage collection triggered by arbitrary free-space thresholds. Instead, garbage collection is synchronously integrated into transaction commit processing, running as incremental tail-to-head relocations bounded directly by the transaction’s time budget. The real-time database kernel continuously tracks execution against transaction deadlines; if an operation risks exceeding its window, the transaction aborts. Crucially, aborting a Copy-on-Write transaction instantly discards uncommitted pages and resets pointer state, naturally eliminating garbage without incurring additional cleanup work. 

Concentrating wear near the head pointer is a non-issue because “head” and “tail” are not static physical blocks, but dynamic pointers sweeping sequentially through a circular physical block space. Moving valid pages from tail to head continuously redistributes active write operations across the entire medium. Every block eventually transitions from head to tail, where stale pages are reclaimed and live pages are swept forward, effectively unifying garbage collection and wear leveling into a single, deterministic mechanism. 

About half of the 4x I/O reduction demonstrated by xflash over conventional stacks comes from structural consolidation: combining the database commit log and FTL address mapping into control pages eliminates redundant metadata updates and double-writing. The other half comes directly from TFTL’s log-structured page reclamation, which dramatically lowers write amplification compared to traditional FTLs that trigger heavy, unpredictable block copy-erase cycles. 

For safety-critical domains, TFTL’s primary advantage is its algorithmic simplicity. For safety-critical systems, the selling point of TFTL is that its garbage collection is algorithmically simple and explicitly bounded by the transaction deadline, which makes it amenable to standard WCET and formal analysis techniques 

Q5.Your earlier work on AGIGARAM NVDIMMs (DRAM + NAND flash + ultracapacitor) demonstrated in-memory database persistence without performance penalty. Now with TFTL enabling deterministic NAND flash access, we’re seeing convergence from both directions: making volatile memory persistent, and making persistent storage deterministic.

Looking forward:

• How do emerging persistent memory technologies (Intel Optane’s successors, resistive RAM, phase-change memory) change the database architecture landscape? Will TFTL-like approaches remain relevant, or does byte-addressable persistent memory eliminate the need for FTL-style indirection layers entirely?
• For embedded systems constrained by power budgets, how do you evaluate the tradeoff between NVDIMM approaches (which require ultracapacitor overhead) vs. TFTL-optimized NAND (which requires more complex software but simpler hardware)?
• More philosophically: As the performance gap between DRAM and persistent storage narrows, do you envision a future where the distinction between “in-memory databases” and “persistent databases” becomes meaningless, or will there always be fundamental architectural differences driven by the physics of storage media?

Byte-addressable persistent memory—such as ReRAM or PCM—theoretically eliminates the need for FTL-style indirection by allowing direct, fine-grained updates without erase-before-write constraints. However, TFTL-style architectures will remain vital due to stark capacity and cost realities. Persistent RAM remains significantly more expensive per gigabyte than raw NAND flash. For any embedded system managing nontrivial data volumes, cost-per-bit dictates that NAND will remain the primary persistent medium, making deterministic FTL management indispensable for bounded execution times.

That said, we don’t believe it’s a relevant tradeoff evaluation. If you need the performance of an in-memory database and require it to be persistent, then some form of non-volatile RAM is your only option. Flash cannot, and will not ever, be as fast as RAM. It’s simply physics. The gap has narrowed and will narrow, but never close. That said, the complexity of any FTL is not visible to application/system developers, and so also shouldn’t be a factor in that evaluation.

Yes, there will always be a distinction, but as the performance gap narrows (even if it can never be closed), it becomes more and more of a niche case where the superior performance of NVDIMM really matters.

Q6. Anything else you wish to add? 

There is another dimension to TFTL that the whitepaper hardly touched on (but alluded to in answer #4 above), and that is lower write amplification as a by-product of combining flash management and database transaction processing. How much lower depends on the workload, flash utilization and other parameters, but this is what we see in our measurements. Lower write amplification translates into better performance (fewer physical writes) and fewer program-erase cycles and longer device life, and potential bill-of-materials cost savings per device. We will be publishing more data on this in the near future.

(*) Real-time Database Management on NAND Flash Storage

……………………………………………………………………

Steve Graves, CEO McObject

Mr. Graves co-founded McObject in 2001. As the company’s president and CEO, he has both spearheaded McObject’s growth and helped the company attain its goal of providing embedded database technology that makes embedded systems smarter, more reliable and more cost-effective to develop and maintain. Prior to McObject, Mr. Graves was president and chairman of Centura Solutions Corporation, and vice president of worldwide consulting for Centura Software Corporation (NASDAQ: CNTR); he also served as president and chief operating officer of Raima Corporation.  Mr. Graves is a member of the advisory board for the University of Washington’s certificate program in Embedded and Real-time Systems Programming. For Steve’s updates on McObject, embedded software and the business of technology, follow him on LinkedIn.

Sponsored by McObject