“Deep Edge” AI: A Conversation with David Such
Q1. Your book (*) focuses on moving beyond just training a model to the complete engineering process of “Deep Edge” AI. For an embedded developer who is completely new to machine learning, what is the single biggest mental shift required when transitioning from traditional logic-based programming to data-driven AI on resource-constrained devices?
It is all about the data. Embedded developers are used to the tight coupling between hardware and software, and adding AI/ML introduces a new dependency. The quality of your ML model is based on the quality of the data used to develop it. From an engineering perspective the model needs to be just good enough to do the job it was designed for. Any more than that is over-engineering and adds cost. Statisticians have different objectives, they want a model that represents reality. Engineers want one that predicts battery charge from noisy readings and fits within the hardware and power constraints of the device. As George Box says, “All models are wrong, but some are useful.”
The first hot word model I trained detected 96 percent of real wake words but had a 55 percent false acceptance rate. In this situation you have to make sure you are optimising for the right parameter. The problem is almost never the model, and in that project the fixes were all in the data. But there are diminishing returns at some point. You have to learn when to stop improving the model and start engineering around it. There is a scaling result I quote from Thompson and co. To improve a statistical model by a factor of k you need at least k-squared more data, and compute scales as at least k to the fourth power. So halving your error rate can cost more than 500 times the compute. If you are at 90 percent and you need 99, you are not as close as you might expect and you probably need a different dataset, a different approach, or a different sensor.
The other shift is harder to get your head around. Traditional firmware is deterministic, by which I mean that given the same inputs it will keep giving the same result. This means with sufficient testing you can have a level of confidence about how a system will perform. A deployed model is probabilistic, and its performance degrades over time. One study I cite states that 91 percent of production ML models lose accuracy over time. Sensors age, environments change, or a part fails and gets substituted. The engineering job does not finish at deployment. You need to think about how you maintain the ML model over the product lifecycle.
Q2. With more than 25 hands-on projects in the book, you cover everything from audio processing to computer vision. When working within the extremely tight memory and processing limits of hardware like the Arduino UNO or Raspberry Pi Pico, what are the primary engineering trade-offs you have to make to ensure the tensor arena doesn’t run out of space?
I define Machine Learning as any method that fits its parameters to data. By that definition linear regression is machine learning, and so are decision trees, support vector machines, and Gaussian process regression. There is a spectrum of machine intelligence from classical algorithms through to deep learning. Where neural networks are useful is with problems that classical methods find difficult.
The tensor arena part of the question assumes that a neural network is the solution and I would argue that often it isn’t in an embedded environment. An algorithm is deterministic, testable, and runs in kilobytes. Half of the book uses classical ML methods for this reason. In the sensor fusion chapter I compare six orientation filters against each other using IMU data, and the most consistent performer was Mahony, a nonlinear complementary filter. You would never replace that with a neural network.
The battery project in Chapter 3 is another example. The lookup table with interpolation was the baseline solution. I compared it against a Gaussian process regression model on the same test data, and the GPR model had roughly 80 percent lower RMSE. The reason is that state of charge is not just a function of voltage. It is a function of voltage, temperature, load, and history.
However, engineering is all about compromise. The battery GPR model would not run on an UNO, you could use it on a Raspberry Pi. For the Arduino I would have had to fall back on a decision tree or a linear model and lose some accuracy. When you do use a neural network and the tensor arena is the constraint, the trade-offs are what you might expect: quantise to int8 (going from float32 to int8 cuts the model size by a factor of four
), fix the tensor shapes at training time so the static allocator can plan, register only the operations the model uses, and size the arena from what AllocateTensors() reports rather than from an estimate. On the Pico the quantised GAN generator needed a tensor arena of 3,340 bytes and ran happily in 4 KB.
The decision process I use is:
- Is there a deterministic solution? If so use that, if not;
- Can I acquire the data to train a model?
- Will the trained model fit and run on the hardware?
In practice the answer is rarely one or the other: often the learned model handles pattern recognition and deterministic code is used as a guardrail.
Q3. You mention that adding AI to embedded systems means knowing what to do “when the breadboard is flaky, the sensor data is noisy, or the tensor arena is too small.” What are some of the most common real-world edge cases or failure modes that developers encounter during deployment, and how does your framework help them troubleshoot them?
I think there are a few common failure modes and they can all look similar from the perspective of the serial monitor.
The first group is a hardware problem pretending to be a software problem. A model that worked on the bench stops working in the field because the supply sags when the radio transmits, the sensor shares a noisy ground with a motor, or a breadboard jumper has an intermittent contact. It is a lot easier to debug software so that is where people focus their efforts. My rule is that if the error is intermittent, start looking at the power rail. Measure, do not assume. And if you cannot be bothered measuring, the answer is bypass capacitors. It is always bypass capacitors.
The second group is the gap between training and deployment. The model was trained on data from one sensor at one sample rate, with preprocessing written in Python, and it is now running on a different revision of the board with preprocessing rewritten in C, using a different sensor. Any mismatch in scaling, normalisation constants, filter coefficients, or window alignment degrades accuracy without producing an error. Quantisation adds its own version of this: an int8 model has a zero point and a scale on every tensor, and if the firmware forgets to apply them, the outputs are wrong. A lot of “the model does not work on the device” complaints are actually “the pipeline on the device is not the pipeline it was trained on”.
The third group is resource limits, and at least these are usually fairly obvious. The tensor arena is too small, an operation is not registered, a tensor has a dynamic shape the allocator cannot plan for, or inference takes long enough to trip a watchdog or starve an interrupt. This is where the engineering magic happens and fixing one problem affects something else. Shrink the model and accuracy drops. Keep the accuracy and inference takes longer. Speed it up with a bigger arena and you are out of RAM. It is like the old joke: memory, latency or accuracy, pick two.
The fourth group is the software you did not write. It comes in two flavours. The first is the library that did not read the data-sheet properly. The second is the library that was never designed for your hardware in the first place. Most embedded ML frameworks and sensor drivers are built and tested on a handful of reference boards, and porting them to anything else means you spend more time on the tooling than the product. The fixes are usually small, a conditional compilation guard here, a stubbed pin there, but they live inside the library, so the next update overwrites them. I write down what was changed and why, because six months later I will not remember. An example from the book is getting TensorFlow Lite Micro running on the Arduino Nicla Vision which required patching four files in the core library.
The approach I use is the same for all of these problems: verify one subsystem at a time, in the order the data flows. Confirm the sensor is producing what you think with a known physical input. Capture the raw stream and run it through the training pipeline on the desktop to confirm the firmware preprocessing matches. Run a single known input vector through the model on both sides and compare the outputs bit for bit. Only then look at accuracy. Debugging an embedded AI system by looking at the final prediction is like diagnosing a car by listening to the exhaust note. When in doubt get out the oscilloscope, a logic analyser makes everything look perfect.
Q4. Extensibility and hardware diversity are major themes at the edge. You utilize Python with TensorFlow alongside the Arduino IDE and Raspberry Pi Pico SDK. How do you see the tooling ecosystem evolving for embedded AI, and should developers look toward unified frameworks or continue to master specialized, vendor-specific SDKs?
I think you need both. The book deliberately uses the Arduino IDE, the Pico SDK with CMake, Edge Impulse, MEMS-Studio for the ST machine learning core, and Python with TensorFlow. Each one demonstrates a different aspect of the problems, and I wanted readers to understand the mechanism before they use a wizard.
The unified frameworks are improving quickly. TFLM, Edge Impulse and CMSIS-NN already cover a lot of ground, and the constant innovation of hardware manufacturers make a hardware abstraction layer very useful. The counter-argument is that the last ten percent of performance on a constrained part almost always comes from something vendor-specific: a PIO block on the RP2040, the machine learning core inside an ST IMU, the NDP120’s requirement that the negative class sort last.
To start I would suggest learning one framework and one SDK. By framework I mean the end-to-end tools that take you from labelled data to a running binary. Examples include: Edge Impulse, or TensorFlow Lite Micro. They are worth learning properly because they get a working prototype onto hardware quickly. By SDK I mean the layer underneath: the Pico SDK with CMake, STM32, the raw TFLM C++ API, or whatever sits closest to the silicon you use. Sooner or later you will need this and the two skills reinforce each other. Knowing the SDK makes you a better user of the framework.
Q5. Many companies are struggling to move AI from pilot to production due to underlying infrastructure and data bottlenecks. From your perspective as a systems engineer, how can teams ensure that data collected at the deep edge is clean and reliable enough to serve as automated training data, like the battery monitor project in your book?
The problem which quickly becomes evident is that the data you need is not sitting on a server somewhere ready for you to download. The battery monitor in Chapter 13 exists because of this. The open-source McMaster dataset for the LG 18650HG2 cell is excellent, but what if you aren’t using that battery? The battery discharge curves published by manufacturers are not useful for training a model which predicts battery state of charge or state of health. You need the measurements behind the curve, taken under the conditions your product will be used in. The best way to get this is to measure it yourself.
However, data collected at the edge is never clean. The goal is data with known problems so you can engineer around them. In order to collect useful data I suggest:
- Logging the context with every sample. A voltage on its own is not that useful for training. A voltage with the device ID, firmware version, sensor type, calibration date, sample rate, ambient temperature and a proper timestamp is. Most data cleaning is an attempt to recover information that should have been logged. You don’t always know in advance which variables matter. A voltage on its own is a poor predictor of battery state of charge. Voltage with temperature, load current and recent history is better, and that is something the model works out during training – but only if you supply the data.
- Using the calibration during data collection that the shipped units will have. Training on lab-calibrated sensors while deploying with factory defaults is not ideal. Every sensor has its own bias and scale error, so either calibrate each device at manufacture and store the parameters (so every device is corrected to the same reference before data reaches the model), or train on enough units that the model learns the spread.
- Monitoring the performance of the model in production and have a way to update it. Things will change and the model will drift, you need a mechanism to correct this. Compare what the device is seeing in the field against what it was trained on.
Q6. You have spent over 30 years in embedded systems engineering, from senior corporate roles at Honeywell and Tyco to founding Reefwing Software. Looking back at your career arc, was there a specific personal “lightbulb moment” or a particular technical challenge that made you realize embedded systems absolutely had to become intelligent, inspiring you to write this 600-page guide?
AI has taken 70 years to become an overnight success. For me. the first lightbulb moment was realising that this was a real pivot point in history. It is hard to know in advance whether something is a fad or a true disrupter but I am a believer. I think AI will be a catalyst for long term change and that makes it worth learning about. The other thing that I am seeing is that intelligence is being pushed to the edge. This is the place where hardware interacts with the physical world and is also where the hardest problems are. Embedded AI is increasingly constrained not by compute but by bandwidth, power, and latency, and moving intelligence to the sensor addresses all three.
In Chapter 9 I use a decision tree to monitor a six-axis robot arm for faults. This runs inside the IMU sensor itself, on ST’s machine learning core.
There are a few reasons that I take a project based approach. The first is that it is more fun and it is the way that I learn concepts best. Because projects deliver measurable outcomes, they demand rigour. You can’t hand-wave a wiring diagram, ignore timing
constraints, or skip model deployment steps – the hardware won’t let you. It also helps me shape what I include in the book. Embedded AI is inherently multidisciplinary. A single project might
span signal processing, machine learning, hardware design, and firmware. That is a lot of ground to cover, so I provided just enough theory to be able to make informed design decisions. At least that was my intention, I do end up down some rabbit holes.Writing this book has changed my view of where the field is going. Nearly all current AI is modelled on the neocortex, the part of the brain that does reasoning and language. But the neocortex was the last thing that evolution built in the brain. Everything that keeps an animal alive sits underneath it: reflexes, balance, drives, and motor control. Those layers run on about 20 watts and are what gives you motivation and purpose. A related issue is that you cannot certify that a probabilistic model will always act in a safe manner. That is why self-driving cars put deterministic guardrails around the neural network. The model is used for pattern recognition but then a separate controller, that can be tested, decides what it is allowed to do.
Biology solved this same problem. The reflexes that keep an animal safe sit underneath the layer that thinks, and have priority. That is the architecture I think embedded AI needs to copy.
Qx. Anything else you wish to add?
One observation about AI and coding, since I am experiencing that change first hand. I have mixed emotions about this. On the one hand, this will allow far more people to develop code, and it will probably work for what they need. The models now are better and much faster coders than I am. On the other hand, coding was an activity that I enjoyed. I would argue that learning to code is still important. It is hard to review code that you don’t understand. And you cannot ask for something that you do not know exists. The skills will shift toward architecture, design and testing, but those skills are built on having written a lot of code yourself.
…………………………………………..
David Such is an embedded systems engineer and founder of Reefwing Software, where he builds IoT devices, robotics platforms, and drone flight control systems. He has over 30 years of experience in embedded development, including senior roles at Serco Australia, Honeywell, and Tyco. His technical writing on embedded AI has a substantial following among hardware engineers working at the edge.
Resources
(*) Embedded AI
Intelligence at the Deep Edge
by David Such
September 2026, 600 pp. ISBN-13: 9781718504905
No Starch Press, Inc