From Replication to Resilience: Miguel Araújo on Building the Operational Layer of MySQL High Availability
Q1. As the tech lead for AdminAPI, you’ve spent years helping shape how MySQL users deploy and manage InnoDB Cluster, ReplicaSet, and ClusterSet environments. From your experience, what are some of the most important design and operational considerations teams should understand when deploying InnoDB Cluster in production, and how does AdminAPI help simplify those tasks while still supporting different operational models and requirements?
I tend to split it into two parts: infrastructure and day-to-day operations.
On the infrastructure side, InnoDB Cluster is a distributed system, so you are immediately dealing with real-world network behavior: latency, packet loss, and transient network partitions. This is where the classic fallacies of distributed computing stop being theoretical. Distinguishing a dead server from one that is simply slow or temporarily unreachable is surprisingly difficult, and in production, if something can go wrong on the wire, it eventually will. Distributed systems therefore need to be designed with those realities in mind, including measures such as protecting quorum and placing members in independent failure domains.
But architecture alone does not deliver high availability, operational readiness matters just as much. Initial deployment is only the beginning. The harder questions arrive on day two: How do you replace a failed server at 3 AM? How do you perform rolling maintenance, recover after an outage, or add capacity without losing quorum or availability? Those workflows determine whether a deployment is genuinely resilient or merely looks highly available on paper.
When we started designing AdminAPI, we deliberately did not begin with replication protocols or low-level mechanics. We asked a much simpler question: What is a DBA actually trying to achieve?
DBAs do not wake up wanting to execute low-level replication commands. They want to deploy a cluster, replace a failed server, perform maintenance, recover after an outage, or scale the deployment. They should be thinking about those goals, not about every mechanical step required to achieve them. Most of those mechanics should be handled by the management layer.
That became the foundation for AdminAPI. When you add a server, for example, AdminAPI handles the heavy lifting behind the scenes: validating the instance, checking its configuration and GTID history, guiding or automating the choice between incremental recovery and Clone provisioning, configuring the instance, and keeping the cluster metadata consistent.
We also designed it knowing that people would use it in different ways. Some administrators work interactively in MySQL Shell, while others embed AdminAPI in deployment automation or build higher-level control planes around it. The MySQL Operator for Kubernetes is probably the clearest example.
What I am most proud of is that the same operational model expanded beyond InnoDB Cluster to InnoDB ReplicaSet, ClusterSet, Read Replicas, and Routing Guidelines, without forcing users to relearn everything. Perhaps Lindy’s law applies here: the longer these abstractions remain useful and adapt to new architectures, the more confident I feel that we got the fundamentals right.
Q2.MySQL Router plays a key role in InnoDB Cluster and ClusterSet deployments by providing transparent routing and failover capabilities.What are some of the key concepts practitioners should understand around metadata caching, routing behavior, and failover handling to ensure predictable behavior and high availability in production environments?
Router’s role is deliberately focused. It needs to learn the topology, apply the configured routing policy, and react predictably when things change.
One of our early architectural decisions was to keep that boundary clear. Responsibilities such as distributed consensus, primary election, and topology management remain with Group Replication and the management layer. Router has a much more focused job: it consumes authoritative topology information and gives applications a stable, predictable way to connect to the right server.
That is where the metadata cache comes in. It combines topology metadata with the current Group Replication state to understand which instances are available and what roles they have. The refresh strategy also influences how quickly Router reacts to changes. Poll too frequently, and every Router continuously queries the cluster; poll too slowly, and failover detection takes longer. When enabled, Group Replication notifications help Router refresh its view proactively instead of simply waiting for the next polling interval.
For routing behavior, it is important to distinguish between basic connection routing and read/write splitting.
With the traditional read-write and read-only routes, routing happens at the connection level. The application chooses the endpoint that expresses its intent, and Router selects a destination when the connection is established. From that point, it forwards the protocol traffic without inspecting each SQL statement.
Read/write splitting adds another model. There, Router classifies statements and can use both a read-write and a read-only destination within the same client session. In current releases, bootstrap creates a read/write-splitting route by default unless it is explicitly disabled. That gives applications a simpler interface, but it also means practitioners need to understand the consistency and session semantics of splitting reads and writes across backends.
We also realized that no single routing policy fits every deployment. That is why Routing Guidelines provide a declarative way to express routing intent without making Router itself increasingly complex.
Failure handling requires a similar distinction. Router does provide connection failover, but it does not perform database role failover. In InnoDB Cluster, Group Replication detects failures and elects a new primary. In ClusterSet, a switchover or emergency failover changes which cluster is primary. Router observes the resulting topology and routes connections to eligible destinations.
With traditional routing, existing connections to a failed server will still be lost because Router cannot preserve an in-flight transaction or migrate a live session. Applications therefore still need sensible reconnect, timeout, and retry logic. Transparent routing gives applications a stable endpoint as the topology changes, but an established session is not transparently migrated when its server fails.
Operationally, Router should be treated like any critical HA component: deploy it redundantly, monitor it, and test it as part of the complete failover path. Over the years, I have come to believe that a routing layer should be almost boring; predictable behavior is far more valuable than cleverness.
Q3. ClusterSet enables organizations to extend MySQL high availability across multiple clusters and locations. What are some of the key architectural tradeoffs and operational considerations teams should evaluate when designing a ClusterSet deployment, and how does AdminAPI help manage the added complexity of these distributed topologies?
When we were designing ClusterSet, one thing became very clear: disaster recovery forces you to separate two problems that are often blurred together: routing traffic and protecting data.
ClusterSet is not intended to replace InnoDB Cluster, it addresses a different tier of availability. Within a region, InnoDB Cluster guarantees zero data loss (RPO = 0) and quick failover. ClusterSet extends that architecture across regions or sites for disaster recovery. Because replication between clusters is asynchronous, teams accept the possibility of some data loss and a longer recovery time in exchange for surviving a complete site failure. The benefit is that inter-region latency does not sit on the normal write path.
That tradeoff has practical consequences. Teams need to monitor transaction-set freshness together with transport and apply lag. Missing transactions determine the potential RPO, while unapplied backlog can also affect promotion time and RTO. Replica clusters must have enough capacity to carry production traffic after promotion, and failover, recovery, and failback procedures should be tested regularly.
The trickiest scenario is a network partition. Cross-cluster disaster failover is deliberately an operator decision, not an automatic election. If the original primary is unreachable, it may still be online and accepting writes, while some Routers may retain a stale view of the topology. Promoting another cluster under those conditions can leave two primaries temporarily coexisting with divergent transaction histories.
That led to a core design principle: automation should never silently convert an operational failure into data loss. During an emergency failover, AdminAPI invalidates the old primary in metadata, but invalidation is not physical fencing. Operators must still isolate or shut down the old primary. Once it becomes reachable, AdminAPI provides explicit fencing and guarded repair or rejoin workflows. If AdminAPI cannot establish that a cluster is safe to rejoin, it should not guess.
That is also why AdminAPI distinguishes sharply between a planned switchover and an emergency failover. Both change the primary cluster, but their consistency guarantees are fundamentally different. A switchover preserves consistency; an emergency failover restores service while explicitly accepting the possibility of data loss.
ClusterSet does not remove the hard tradeoffs of disaster recovery. AdminAPI makes them explicit and provides predictable workflows for navigating them safely.
Q4. You recently presented the integration of MySQL REST Service directly as a MySQL Server component. Can you explain the technical reasoning behind this architectural approach, the benefits it provides from both performance and security perspectives, and the new opportunities it creates for developers building modern applications on MySQL?
When we started working on MySQL REST Service, the first big architectural question was where it should actually live.
Starting with Router was a deliberate choice. Router already had a mature HTTP runtime, integrated naturally with our HA deployments, and could scale independently. But for single-instance deployments, rapid prototyping, or lightweight containers, running a separate Router process can be unnecessary overhead. Embedding MRS into the server was meant to complement the Router model, not replace it.
One thing I felt strongly about was avoiding a second HTTP implementation inside MySQL. Instead, we refactored Router’s HTTP runtime into reusable libraries and embedded those libraries into the server. Both deployment options therefore use the same underlying implementation.
The bigger change was how we thought about MRS. In component mode, it is no longer an external client. Normal metadata queries and application SQL run through the server command service, using embedded sessions under dedicated security contexts rather than outbound client connections.
Embedded sessions eliminate the need to manage passwords or keyring entries for internal execution identities while preserving MySQL’s authorization and privilege checks. As a component, MRS also integrates with the server’s logging, configuration, and lifecycle management. HTTPS is provided by the embedded Router HTTP and TLS stack through a separate TLS context.
Removing the extra process boundary and normal SQL network path can reduce overhead, but I think the larger benefit is operational. Developers can expose REST endpoints from a single MySQL Server process and provide applications with HTTPS and JSON access without building a custom REST layer for every data-centric use case. It does not replace application-specific business logic, but it can eliminate a lot of repetitive plumbing. Larger deployments can still use Router for independent scaling and HA integration.
That is the kind of outcome I always hope for: new functionality should feel like it belongs, not like something that was tacked on afterward.
Q5. You began your career designing a fault-tolerant replication mechanism for MySQL that enabled massive scalability with a large number of replicas — work that culminated in your Master’s thesis on database replication in large-scale systems. Since then, MySQL has evolved from traditional replication to technologies such as Group Replication, InnoDB Cluster, and ClusterSet. Looking back, what technical advances in replication and high availability have had the biggest impact on real-world deployments, and what areas of distributed database infrastructure do you think still present the most interesting engineering challenges?
When I started my Master’s research, I was looking at how to scale replication to a huge number of replicas without turning the primary node into a massive bottleneck.
Back then, I built a prototype using Spread, a group communication toolkit that was pretty popular at the time. The idea was to distribute changes efficiently and guarantee fault-tolerant delivery through group communication rather than relying purely on direct one-to-many replication. At the time, I definitely didn’t imagine I’d eventually get to tackle those exact same problems inside MySQL itself.
Years later, seeing Group Replication bring so many of those concepts into production was immensely satisfying. The actual implementation was very different, of course, but the core building blocks like group membership, ordered communication, failure detection, quorum, and coordinated recovery, were fundamentally the same.
Looking back across MySQL’s evolution, GTIDs were a huge turning point because they made automatic positioning, failover, and topology tracking drastically easier to automate. Group Replication then baked high availability right into the replication layer itself, and AdminAPI and ClusterSet completed that path by turning low-level mechanisms into clean operational workflows. That last part is critical, because a solid replication protocol is only half the battle. DBAs still need safe, realistic ways to manage, repair, and scale a live topology.
What strikes me is that the hardest engineering challenges today are still the classic distributed systems problems. The CAP trade-offs haven’t gone away: during a network partition, you still have to choose how to balance consistency and availability. Cross-region latency remains a fundamental characteristic of distributed systems that software must work with rather than eliminate. Systems still need to handle transient failures gracefully, avoid performance hotspots, and stay predictable during recovery.
Sixteen years ago, I thought database replication was mostly about moving data around. Today, I see it as building distributed systems that stay safe, correct, and predictable when everything around them is failing.
Q6. In addition to your work on AdminAPI, Router, and REST Service, you’ve also contributed to MySQL’s component infrastructure. For developers who may not be familiar with this area of the server, what role do components play in the evolution of MySQL’s architecture, and how do they enable new functionality to be developed and integrated more effectively?
Before I worked on MySQL REST Service, I understood the Component Framework mainly from an architectural perspective. After building a fairly complex server component on top of it, I came to appreciate it on a completely different level. Extensibility itself is not new to MySQL. Major capabilities such as Group Replication and the X Plugin demonstrated that substantial functionality could be added without putting everything directly into the server core.
What the Component Framework changes is how new functionality interacts with the rest of the server. Instead of reaching into private implementation details, components consume well-defined services and declare their dependencies explicitly. In a codebase as mature as MySQL, tightly coupled subsystems make long-term evolution increasingly difficult. Components give you cleaner boundaries and clearer contracts.
Working on MRS made that very concrete for me. Adding a feature to a database server is rarely just about implementing the feature itself. The harder part is making it behave like a native part of the platform. A component can use shared server infrastructure for lifecycle management, configuration, logging, and security without depending directly on private core internals.
Ultimately, the most interesting shift is mental. Instead of engineers asking, “Where do I modify the server?”, the question becomes, “What service should the server expose?”. That encourages a much healthier separation of concerns and makes new capabilities easier to develop, test, integrate, and evolve over time.
To me, that is the real architectural value of the Component Framework: it allows MySQL to keep evolving without turning every new capability into another permanent dependency on the server core.
Q7. Looking at the MySQL Community Roadmap, which new proposed features are you most excited about from an engineering perspective, and why?
Since the roadmap is still being defined via public discussions and contributor summits, I view these as proposed features and priorities. Many of them are already under development, and you can view the discussions and issues as they progress on GitHub. From an engineering perspective, there is a lot to be excited about.
If I had to pick one, it would probably be the continued evolution of the Component Framework. One of the public proposals discusses exposing SQL data types, SQL operators, index access methods, and optimizer hooks as component services. Having worked on the component infrastructure recently, I find that direction especially compelling because it changes not only what MySQL can do, but how the server itself can evolve.
Native vector storage and VECTOR INDEX support are another major area. MySQL already has a VECTOR data type, but the proposal goes much further: distance and similarity functions, ANN indexing and search, and a fully functional, ACID-compliant vector store. What I like about that approach is that it supports modern AI use cases while treating vector search as a first-class database capability. It is deeply integrated with the existing transactional, security, backup, and replication infrastructure rather than added as a standalone extension.
The Hypergraph Optimizer has already taken an important step by becoming available in MySQL Community Edition. What interests me now is the continued work on performance and feature parity: better cardinality estimation, costing, join ordering, access-path selection, plan alternatives, and more structured diagnostics. Query optimization remains one of the hardest database engineering problems, and improvements there can affect almost every non-trivial workload.
Given my background, replication and binlog modernization naturally catch my attention as well. The public proposals include streaming changes through the replication pipeline earlier, improving batch and group-commit processing, modernizing the binary-log infrastructure, adding richer metadata, and building stronger CDC frameworks. Those are deep infrastructure changes, but they address very real throughput, latency, recovery, and integration bottlenecks in large production environments.
What I like most about the roadmap is the balance. There are visible new capabilities, but they are supported by foundational work on extensibility, optimization, replication, and data infrastructure. Those foundations often matter more over time than any individual feature.
Qx. Anything else you wish to add?
One of the reasons MySQL became so popular is that it made databases approachable. It was powerful, but it was also practical, you could install it, understand it, and start getting useful work done quickly.
With AdminAPI, that is what we tried to bring to high availability. Distributed systems are inherently complicated, but operating one should not require every DBA to be an expert in consensus protocols. I am genuinely proud that we managed to make those deployments more accessible without pretending that the underlying complexity does not exist.
I am also very glad to see MySQL becoming more transparent through public roadmaps, open design discussions, and earlier community feedback. I hope that encourages more people to participate, whether by submitting a bug fix, improving documentation, testing a new feature, suggesting an idea, or using the component infrastructure to build something new.
I also think AI tools can make that first step less intimidating. I find them useful for finding my way around an unfamiliar part of a large codebase, sketching an initial test case, or challenging an assumption. They do not replace engineering judgment or code review, but they can reduce the effort required to get started.
More than anything, I would like to see more people get involved and help carry that original MySQL spirit forward. MySQL has always benefited from a strong community, and I hope that continues for many years to come.
Editor’s note:
What is a failure domain?
In modern cloud computing environments, infrastructure resources are typically organized into a hierarchy of Regions, Availability Zones, and the servers (“compute instances”) themselves. When designing services from an availability point of view, these three “levels” are more abstractly known as failure domains. Within the same failure domain, many types of failures could affect also other services inside the same failure domain. For example, if a server loses network connection, it is possible that the servers right next to it have lost network connectivity, because they are all connected to the same router.
It’s the same principle as not putting all your eggs in a single basket. Given multiple baskets, the only question remaining is, how do you distribute your eggs into the baskets, to[HV1] ensure that at least the minimum amount of eggs needed will make it to the final destination. And that is the principle of how you architect robust high availability solutions such as InnoDB Cluster.
……………………………………………………
Miguel Araújo
Senior Principal Software Engineer at Oracle MySQL
Miguel Araújo is a Lead Principal Application Software Engineer on the MySQL team at Oracle, where he works on the architecture and development of solutions for high-availability, replication, and scalability. He is the technical lead for MySQL Shell’s AdminAPI, the core component that drives MySQL InnoDB Cluster, InnoDB ReplicaSet, and InnoDB ClusterSet. He also leads the development of the MySQL REST Service (MRS) component, which exposes MySQL data and functionality through REST APIs. In addition, he contributes to MySQL Router, which provides application connectivity across these topologies, and to the MySQL Server component infrastructure, which enables modular server components and services. Miguel holds a degree in Computer Science Engineering and a master’s degree from the University of Minho in Portugal, where he conducted research in distributed databases.
As part of that work, he designed a fault-tolerant replication mechanism for MySQL capable of supporting large numbers of replicas, which became the basis of his master’s thesis. Since joining the MySQL team at Oracle, Miguel has worked on MySQL Enterprise Monitor, MySQL Proxy, middleware, client technologies, and high-availability solutions. Over time, his work has evolved from developing individual components to helping shape the architecture and technical direction of MySQL’s high-availability and developer-facing technologies. Miguel cares deeply about usability and believes that distributed systems should not only be resilient and technically robust but also be intuitive to deploy and operate. He is based in Portugal.
Profile and links: about.me/miguel-araujo
Sponsored by MySQL/Oracle