Tech Stack Complexity and the Ever Worsening Signal-to-Noise Ratio
By David Rolfe, VoltDB’s senior technologist in EMEA
I’ve lost count of the number of times I’ve heard someone brag about how they got a set of disparate components to work together, but I almost never hear (or read) about someone making something simpler and, in doing so, saving money.
The fact is, the sheer complexity and variety of what people are deploying these days is driving up costs and reducing the focus on business value, an issue that deserves greater attention and discussion, especially because there are now many ways to simplify your tech and reduce costs.
The TL;DR is that the development community needs to pay more attention to the needs of the business and less attention to stitching together wildly ambitious but incredibly costly tech stacks composed of disparate, and ultimately incompatible, solutions.
This can also be described in ‘signal-to-noise ratio’ terms:
Actual business value that generates money is ‘signal’; everything else is ‘noise’ and is something we tolerate to get ‘signal’. Without ‘signal’ ( ‘business value’) the ‘noise’ (everything else a developer does) will sooner or later stop.
But how did we even get to this point?
Let’s discuss.
How Everything Became So Complicated
About half the job of anyone in software development is fully understanding what your employer is trying to do. This is challenging because not only are some domains inherently complicated, but the enterprise itself may not understand them to the level of detail required to ‘write code for fun and profit’. If you don’t know what the problem is, you can’t solve it.
In order to handle this complexity, any tool that tries to take the hard work out of your hands will inevitably become more complex itself, until it starts to look suspiciously like a computer language hiding inside a development tool.
The fact is that business complexity is here to stay, and sooner or later developers will need to understand the eccentricities of the problem space they operate in.
For example: At one point in my career, I may have inadvertently become an authority on the criteria for obtaining UK government grants to local authorities to replace missing cats eyes on regional roads. This sounds, and is, wonderfully obscure, but I’m fairly sure that it wasn’t until I sat down and tried to write a report that maximized the grant potential for a given stretch of asphalt that anyone really, really looked at the regulations in forensic detail, with the goal of maximizing grant size. Bear in mind that the replacement costs for all the roads in a single county in the United Kingdom has been quoted at GBP24 Billion, so apparently trivial changes in interpretation can have monumental financial implications.
I know I’m not the only person to have experienced this kind of hyper-specialization.
We’ve seen massive change over the last two decades, which has been accompanied by a dramatic increase in complexity that has stymied any improvements in our signal to noise ratio. It used to be that to solve a problem we’d all go into a dark room with a white board and not come out until we knew exactly what we were going to do with our two servers, one of which was an app server and the other a database server. If we couldn’t get it to run on two servers we’d figure out why, and if we absolutely had to we’d buy a third or replace one with a bigger model.
Since then, an amazing array of open source projects and hyperscaler products have unarguably made the world a better place by giving us hundreds of ways to solve problems. But it’s also created a veritable Tower of Babel when it comes to languages, paradigms, and stacks. I’d argue that it’s simply not possible for an individual developer to know all of these properly and simultaneously maintain knowledge of the business challenges that affect his employer’s bottom line.
Not only has choice proliferated beyond the ability of a single developer to understand it, the barriers to entry have been lowered. It used to be that I’d have to find a spare server, download binaries, resolve dozens of dependencies and then start using something. This forced me to justify the need. Now I can go into my hyperscaler console and ask for one.
In fact as a 2022 developer, I can spin up a whole series of services and start making money in a few minutes. Unfortunately, there is no guarantee I’ll be making money for my employer instead of the hyperscaler hosting the services.
Add Latency to the Mix
Another issue emerges at this point. Not only does the proliferation of components increase ‘noise’ and reduce ‘signal’, but each boundary between a stack layer introduces latency and complexity. Components that behave well individually start to misbehave when you stack them, simply because they were built by different people with different perspectives and expectations. This further harms the signal to noise ratio.
When you add microservices to the mix, things can start to get out of hand. While it might be superficially simpler to replace a monolithic entity with a series of smaller components or microservices, each time you split a component into two you create a new boundary, new and unhelpful latency issues, a new API, and a new set of potentially exciting failure modes. As a database person, I constantly see people trying to pretend that state and its related issues don’t matter when you create a microservice. Instead of meeting the needs of the business they end up re-implementing two phase commit so that their own stack stays consistent.
Then there’s the challenge posed by 5G and edge computing: Both technologies use ‘low latency’ as a selling point, but the kind of convoluted complexity we’ve talked about above is infamous for harming latency.
Conclusion
We’re now surrounded by ‘solutions’ but don’t seem to be delivering any more value per man hour than before and way too many developers willing to try to get them to work. But if you don’t pay attention to the ‘‘signal to noise’ ratio and how it affects business value, you’re heading for trouble.
For software developers, it’s a case of understanding their tools and understanding the business problems they’re being paid to solve. We all know it’s impossible to only have ‘signal’, but the ratio between the two matters..
………………………………
David Rolfe is VoltDB’s senior technologist in EMEA. He has 30 years of data industry experience, half of which has been in the Telco Industry. In prior roles he designed and built charging, policy and mediation systems, managed a globally distributed team of DBAs and oversaw the selection of technology to replace legacy RDBMS. David is also credited as the inventor of four separate patents in the area of real time conflict resolution in distributed databases. David was Oracle’s 6th employee in Ireland and later went on to work at Sun, Pacific Bell, LeapFrog and Openet/Amdocs. David is based in Dublin, Ireland and has three teenage children.
Suggested Resources:
Stack Simplification (Latency Squeezing) – VoltDB
Sponsored by VoltDB