On Kubernetes, Cloud and Couchbase Autonomous Operator. Q&A with Anil Kumar

Q1. Why Kubernetes is important?

When we first started using commodity computing, there were many overheads associated with managing data center power and cooling, maintaining network switches and routers, and then the servers and storage themselves.  Automated orchestration made this more efficient, but in the end, it was the managed cloud that was the real game changer – physical resources replaced by virtual and created with the click of a button.  The benefits were better resource utilization and vastly improved time to market, without the procurement, leading to cost savings.

However, those virtual machines still need management, security policies, auditing, and monitoring.  Kubernetes is the next giant leap forward, delegating these requirements to the cloud operator, leaving the businesses to concentrate exclusively on creating and deploying their application, leading in turn to yet more operation efficiency and cost savings.

Containers in general are a philosophical change for the better.  By breaking down monolithic applications into a microservices-based architecture, individual components become simpler, more reliable, and can be upgraded or patched to mitigate security threats independently of the whole.

Q2. Complexity is a common criticism of Kubernetes. What is your take on this?

Kubernetes may be complex, but not unnecessarily complex.  In fact, the complexity of container orchestration has led to the advancement of Kubernetes as the leader in simplifying the deployment and management of containers.  So relatively speaking, whatever complexity is attributed to Kubernetes is significantly lower than what we would find when attempting to deploy applications at scale with plain old docker or rocket.

Q3. What is the Couchbase Autonomous Operator for Kubernetes and how does it help to reduce complexity, human error and manual management of database services? Please gives us some examples.

The Couchbase Autonomous Operator for Kubernetes is an application-specific controller that extends the Kubernetes API to automate the creation, configuration and management of the Couchbase Data Platform.  The Operator reduces operational complexity up to 95% by implementing the operational best practices that most efficiently deploy and manage the Couchbase Data Platform.  When run alongside the Couchbase Admission Controller, validation of the Couchbase Data Platform is provided to ensure operational best practices are being employed.  For example, it’s possible to configure Couchbase Service memory quotas so that they exceed the total available memory available on a cluster, but the Admission Controller would reject this configuration and notify a user to adjust memory quotas to acceptable values.  Without this preflight validation, the cluster would have been partially deployed and in need of manual support to proceed with deployment.

Q4. What about resource consumption and cloud portability?

While resource consumption improvements are typically thought to come down to physical consolidation and cost reduction, one of the most overlooked is typically people who keep these things running.  By leveraging economies of scale and automation, the big cloud providers can pass on the cost reductions in maintaining the underlying hardware and software.

We provided the Couchbase Autonomous Operator in order to reduce the database administration from the equation and distil all that application specific knowledge into a simple, easy to use application.  We also have a wealth of operation experience from company experts and customers from all over the globe that help shape what the Operator does and more importantly the best practices we recommend.

The Operator is fully certified on Amazon, Google and Microsoft managed Kubernetes offerings along with RedHat Openshift.  The same configurations are tested across all of these platforms so we have a very high confidence that your infrastructure can be ported effortlessly.  Using Couchbase Data Platform features such as cross datacenter replication it is even possible to perform cross cloud migration – live – with minimal down time.

Q5. Why did you extend the Kubernetes API?

The Kubernetes API was extended to provide an operational specific controller capable of managing The Couchbase Data Platform.  Kubernetes ships with many generic controllers for managing standalone Pods, Pod Replicas, Jobs, and DaemonSets, and while these controllers can be used to create single Couchbase nodes, they are not useful for cluster nodes and managing the lifecycle of the nodes.  Therefore, we created a custom controller with the operational knowledge of creating a Couchbase cluster, so that ‘scaling up’ could be translated as ‘add a node to the cluster and rebalance.’

Q6. What basic Kubernetes resource and controller concepts is Couchbase Autonomous Operator using?

The Operator is built upon RedHat’s Operator SDK framework, so benefits from industry standard best practices out of the box and helps us stay aligned with the ecosystem as a whole. Couchbase cluster models are persisted in Kubernetes’ internal database.  We, as a controller, watch for modifications and act upon them, while also monitoring existing resources and ensuring they are in the requested state as described by the model.

The Operator is responsible for creating Pods, PersistentVolumeClaims and Services in order to create the cluster, provide disaster recovery and also connectivity in all common network topologies.

We also provide support for PodDisruptionBugets that allow us to control how the underlying Kubernetes cluster behaves when upgrading.  By using this resource we can control exactly how many pods can be deleted at once and when we allow any more to be deleted.

Q7. What else does it include?

We realize that there are inherent complexities with debugging issues across a relatively large set of resources.  This allows inefficiencies to arise when supporting a customer deployment, the usual manifestation is a lot of back and forth between the customer and support team.  To solve this issue we provide a simple support tool that is able to collect all resource definitions, logs, event streams etc. in a single command.  Where possible we limit the data collected to only relevant objects associated by a Couchbase cluster, or set of them, in order to limit the scope of what we collect to keep your data secure.

Q8. What is your advice: Is it better for an enterprise to use a single-provider, a hybrid cloud or cross-cloud strategy?

As the old saying goes: “don’t put all your eggs in one basket”.  It certainly makes sense not to rely on a single provider.  By keeping your infrastructure platform-agnostic and deploying across multiple providers you can be sure that everything is fully tested and can be migrated from one platform to another as desired by the business.  Add to that the inherent fault tolerance of having providers isolated from each other and it becomes a compelling argument provided you have the necessary content delivery network to ensure uptime across the providers.

There are however inherent problems with this approach.  Take for example security information and event management.  One provider may patch CVEs significantly faster than another so you must take this into consideration.  Also the provider APIs are likely to differ so this again may prove to be a barrier to entry.  The industry will be quick to catch up however providing federated management via single pane of glass interfaces that will certainly make cross cloud simpler.

Q9. As Kubernetes continues to gain momentum and IT organizations increasingly adopt cloud computing and embrace newer technologies like microservices, mobile, IoT and more, what are the key roadblocks that hinder progress?

Cloud, especially Kubernetes, is fantastically successful at deploying and testing stateless applications.  But underpinning all of that you need a database to drive the application and provide analytical insight.  The stateful application has the most inertia to change and is always the last to migrate to new technologies.

The Couchbase Autonomous Operator team is operating right on the bleeding edge of what is possible.  Kubernetes isn’t perfect for stateful applications yet, but it’s getting very close.  We are in a great position to handle all these quirks for you and make data migration a commodity not something to be feared.  We don’t see roadblocks, only opportunities to better the simplicity and efficiency of modern computing.

Qx Anything else you wish to add?

Containerisation and running with Kuberentes is potentially the biggest milestone of the ongoing cloud revolution. At the same time, it lays bare an important point: revolutions will only benefit the business if the entire organisation is brought along. In the world of IT, this means not only rushing to adopt containers and cloud-native applications but ensuring that critical infrastructure such as databases can be brought along for the ride. Organisations that can adapt, stay ahead of the curve, and effectively future-proof their business, will be at a huge advantage.

Anil Kumar [a.k.a Anil Kumar Hampayya Saunshimath] is the Distributed Database specialist and Director of Product Management at Couchbase. Anil’s career spans more than 15+ years building software products across various domains including enterprise software, mobile services, and voice and video services. He is a hands-on product leader responsible for Couchbase Server, Cloud, Containers, and Kubernetes product lines as well as evangelizing the product strategy and vision with customers, partners, developers, and analysts. Before joining Couchbase, Anil spent several years working at Microsoft in the Entertainment division and Windows and Windows Live division. Anil holds a master’s degree in computer science from Toronto (Canada) and a bachelor’s in information technology from Visvesvaraya Technological University (India). He lives in San Jose, California with his wife Suma and their daughter Aarushi.