‘State’ in Kubernetes
Kubernetes has become the defacto application orchestration engine for applications running in containers. Docker pioneered the workflows to build, package and run applications in containers. The first wave of applications that were targeted by Docker and subsequently Kubernetes are stateless applications. What are stateless and stateful application and why is orchestration of stateful application a difficult task? Let us dig deep into some of the core concepts.
What is State
There are several definitions. Here is mine…
- Connection/Request state: DB connection, sessions, HTTP sessions etc
- Process state: In memory state of the process, or groups of processes using shared memory
- Persistent state: The persistent state of the application
The HTTP protocol is stateless by definition. Every request should contain enough data for the server to fulfill the request. HTTP request should not depend on the previous or the next one. Having said that, there are exceptions in the use of the protocol, for example, sessions. Note that statelessness of HTTP means that the protocol itself does not maintain state on either side, the client or the server can save state.
Database connections, however, are a different story. Database sessions and transaction rely on the state. Consider a DB operation using JDBC or ODBC — a connection is opened with the database using user credentials, a session is created, multiple inserts or update statements are executed and then the session is committed or rolled back. All these actions cannot be performed in multiple steps, much less over multiple connections and clients.
Every process has on-disk state and in-memory state. There is of course network state, which the open TCP sessions etc. There are applications built completely on in-memory state — caching services like Memcached or Redis. There are applications that build an in-memory state and then flush the state to disk at periodic intervals like HDFS namenodes, SAP HANA, etc. VMWare vmotion is a solution that can migrate a process state from one host to another. TCP handoff is used to move the open session from one host to another.
The core function of any application is read, process, write data. A process can die, and come back up. When they do, they will need to reconcile with the last state of the applications, which is saved on persistent media. There are several products built around provisioning and protecting this persistent state. The storage industry, backup industry thrive on protecting this state.
Applications on Docker
Let us examine Nginx, the most referred to example for a stateless application. Nginx is an open source reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer, HTTP cache, and a web server (origin server). We will also use Docker to run Nginx server to host our static HTML content. Let us examine the docker command.
$ docker run \
–name my-custom-nginx-container \
# Configuration \
-v /host/path/nginx.conf:/etc/nginx/nginx.conf:ro \
# Data (Content) \
-v /some/content:/usr/share/nginx/html:ro \
Note the bind mounts are used to leverage state from the local file system. We can call the container (a scoped process) as stateless, but the application is not stateless. Again, there is no such thing as a stateless application. There are only stateless processes.
We have identified the first form of state. ‘Content or Data’
Host file system is the keeper of the state. First and the most primitive form of state delegation.
There is no such thing as stateless application, only stateless processes
Applications on Kubernetes
Kubernetes is a very opinionated framework for deploying and managing applications. There are several design elements like POD, Egress, Ingress, PersistentVolumeClaim, PersistentVolume. Applications can leverage these features and in some cases forced to use these primitives. This deserves a whole blog in itself.
PODs, by design, do not retain IP address (Note, this behavior can be overridden). DNS name of the POD will change with every restart. To give a fixed endpoint to the users of the application, one has to set up a service endpoint.
Let us take a look at a stateful application MySQL. I would categorize of MySQL as a simple stateful application.
helm install -name salesdb stable/mysql
This will give us a running MySQL instance. Let us take a look at the composition of this instance? What we notice is that the application is comprised of more components than just a Pod, PVC and Service. There are the basic building blocks that Kubernetes offers to build applications. Let us stick to the concept of state for now as in state of Kubernetes.
‘State’ in this MySQL application
Let us discuss the persistent state. State in data-heavy applications is something that allows for the continuity of the applications. The question that we need to ask ourselves here is ‘What is the dataset that I need for the application to recover after a restart, failure or disaster’
Checklist to help you guide in answering the above question.
- Can the application recover after I restart the Rack?
- Can I take a consistent snapshot of the application?
- Can I migrate my application to a different Kubernetes cluster?
- Can I backup my application to protect against disaster?
PVC (Persistent Volume Claim)
This goes without explanation. PVC represents storage for the application. This is where MySQL database resides. There can be more than one PVCs attached to a POD for workload isolation, performance, and other reasons. All of the PVCs constitute the state.
I think we can all agree that PVCs constitute state.
In Kubernetes, contig maps are the keepers of the application configuration. Additionally, config maps can be used as shared configuration stores. In a multi-tier application, web tier will need to communicate with the database tier and config maps are the way to share the DB settings with the web tier.
If DB starts with some configuration and the web need these configuration settings to access the database, then save just the PVCs is not enough to recover the multi-tier application. Think of SSL. Enabling SSL has to be a configuration setting on both the client and the server end.
Config maps do constitute state
Secrets are like Configmaps that store shared data or supposedly sensitive data like passwords, authentication tokens, private keys, certificates, etc.
When the application is created, the password is in a Secret. Then it gets set in MySQL system tables. Then the web tier will connect to DB using this password. A secret, being a shared data point, has to be ‘state’. If we lose the secret, DB can run fine, but the web server cannot connect to the DB and our application is down.
Secrets do constitute state
This is a surprise. Let us say we have a MariaDB or Percona MySQL clusters. There are more than one MySQL servers running on Master-Slave or Active-Active configurations. There are more than one PVCs. The number of servers, in this case, constitute the state. Is the is a necessary state, may not be. One can always reconstruct the topology but is a very tedious and often error-prone undertaking. In k8s, topology information is captured in Deployments, Replicaset, Statefulset.
Finally, topology information should be considered state
There is no such thing as stateless applications, there are stateless processes. State in an application is always delegated to one or more stateful processes. The state is not just the data volumes, but application configuration and topology information also constitute the state.
Much of what I outlined here are learnings from my extensive work on orchestrating data-heavy application at https://robin.io/
ROBIN brings advanced storage and data management that extends the Agility, Efficiency and Portability of Kubernetes to All Stateful Applications, even complex Big Data, Databases, AI/ML and Custom Apps, on Any Infrastructure, On-Premise, Hybrid Cloud or Multi-Cloud
Ravi Alluboyina is a Senior Software Architect at Robin.io