Using Kubernetes to Modernize Your Elastic Stack

Robin.io Elastic Stack Blog: Part 2

Note: This blog is Part 2 in a three-part series. If you missed the first blog, you can view it here.

Kubernetes is revolutionizing how many enterprises have been managing their bare metal or VM-based Elastic Stacks. Containerizing an Elastic Stack and running it on Kubernetes promises automated deployment, effortless scaling, and cloud portability. It can help developers resolve issues faster, reduce infrastructure costs, and prepare for the cloud-native world.

But the Elastic Stack is a complex, stateful application that was not originally designed to run in containers. As a result, Kubernetes provides some of the basic building blocks for deploying and managing the Elastic stack, but there are a number of unique deployment challenges that remain.

The Elastic Stack was not built as a stateless microservices application. Kubernetes was developed with stateless applications in mind. A stateless app is a program that does not store data generated in one session for use in the next session, so no data is persisted. A common example is a web server. In contrast, a stateful application needs to persist data. The Elastic Stack is a complex stateful application, as it needs to persist data and there are multiple components involved. Provisioning and scaling is relatively easy for stateless microservices, since developers can simply create and add pods. But for the Elastic Stack, provisioning and scaling isn’t as simple since developers must register the new loads with a load balancer to ensure optimal performance.

Provisioning the Elastic Stack requires complex orchestration. It takes a lot of legwork to make the Elastic Stack run properly on Kubernetes, because they were not made for each other. As a result, IT teams have to maintain giant configuration management playbooks for their deployments and reinvent a lot of wheels. While Kubernetes can help in setting up the separate Elastic Stack components and simplify deployment, IT teams need capabilities to help ensure performance and availability of the entire stack. This can only be accomplished with tools that have both application- and infrastructure topology-awareness.  App-awareness ensures performance through proper data locality in the cluster, while topology-awareness handles the optimal pod placement to make sure different components of the Elastic Stack remain highly available.  

Statefulsets and persistent volumes by themselves are not a complete solution. Kubernetes supports storage for pods using persistent volumes. However, Kubernetes by itself doesn’t ship with any storage or data management stack. An external storage stack must be used to provide the actual storage and data management capabilities for the persistent volumes. Because Kubernetes doesn’t have its own storage stack, there is no way to ensure performance SLAs or IOPs guarantees when running multiple Elastic applications on the same Kubernetes cluster, giving rise to the “noisy neighbor” problem. Persistent volumes also do not support data locality, affinity, anti-affinity, and isolation constraints.

Helm charts and Operators are a good start, but do not provide a practical solution. Existing Helm charts and Operators provide some direction on how to create a basic deployment of the Elastic Stack on Kubernetes. However, they still require users to be experts in Kubernetes before they can attempt to deploy the Elastic Stack. Also, these basic deployments that cannot ensure data locality or high availability are not suitable for practical applications. Moreover, if the goal is to create an “as-a-service” experience similar to popular hosted Elastic Stack offerings, IT teams will have to do all the heavy lifting. Helm charts and Operators do not provide the app-store experience and simplified day-two operations needed to create a managed service in-house.  

In order to solve the challenges encountered when running Elastic Stack on Kubernetes, IT teams need the capabilities of a powerful automation and management platform. Stay tuned for Part 3 of our series, where we’ll examine the functionality needed in an effective management and automation platform, with a checklist that can be used when selecting the best solution for your Elastic Stack deployment.