Advice on Scaling Stateful Applications with Containers from Craig Brown, PhD

Of all the up-and-coming technologies that stand to improve enterprise application performance, containers top the list. But when it comes down to applying this new technology in a stateful container model, many enterprise IT teams are left scratching their heads, especially when scaling stateful applications.

While containerizing microservices, proxies, and application code make for fun side projects, that doesn’t crack the bigger nut for most enterprise teams, which is improving the efficiency of stateful applications that can handle robust databases and database application functionality.

Containers are fast to deploy, great for program portability, and make efficient use of system resources. But the vast majority of current enterprise container projects are stateless in their nature.

scaling stateful applications

Which is why we recently called on Craig Brown to help us make sense of how to best apply containers in the enterprise. With more than 30 years of experience in the database realm, Craig is one of the leading thinkers and practitioners working in big data and database applications.

Here’s what he had to say about applying container technology in the enterprise:

Robin: Tell us a bit about your history. What has your experience been in big data and databases?

“I’ve been working within the data arena for the past 30 years. Like many others, I started out in programming and moved up into more complex database work, which I then focused about 20 years on. 10 years ago I shifted focus to big data. At this time, it was really in its infancy stages. Overall, I was looking for a more efficient, and innovative way of dealing with data because frankly, I’ve always been an out-of-the-box thinker. I’m interested in anything that offers alternatives to a traditional way of doing things with databases. I started to bring these innovations to executives, and built my goals around improving speed, scalability, and flexibility.”

Robin: How do you think about stateless and stateful applications? And specifically, how can containers apply to database applications?

“A few years back I would have answered a little differently. I would have said containers are just an advancement over what we typically see, with virtual machines and VM environments. Many professionals were taking a stateless approach by dividing hardware into domains or large partitions in order to spread the processing across multiple entities and improve speed. But, that meant losing crucial data. Today, with the advancements in the container industry, things are moving well beyond virtual machines. Now we’re seeing true benefits of distributed processing. Looking at applications, when you say “stateful” and “stateless” we’re really just talking about the ability of the application to stand alone, grab the data they need, and service communities with features and information specific to the nature of that application.

The overall comparison between stateless and stateful is really a matter of how is one of those scenarios best going to utilize the data associated with what that application is supposed to produce, and the benefits it is meant to provide to the users of that data. Whether talking about stateless – meaning the data can exist, but not have communication with the operating system – or stateful and it needs to talk to other components and operating systems – it’s just a matter of how the application is developed, and ultimately what end purpose it serves.”

Robin: Are there speed or security benefits to building stateful applications?

“There are quite a few benefits. I think the stateful versus stateless approaches really started with microservices, and the ability to break application code down into a very specific piece of functionality and separate it so that it became faster and optimized. But when you start talking about stateful, you need data across those microservices. For example, if you have a single piece of Java code and its sole purpose might be to count something, or perform any other very simple task, stateless is fine from a microservices standpoint. As these services expand, the need to go stateful is probably more prominent because now you need data across sessions, and you need to know what the other services are doing and providing so that efforts are not duplicated. That’s why we’re having this conversation. It’s becoming a far more dominant approach. Stateful is really starting to dominate needs and benefits of complex applications.”

Robin: You recently wrote that in managing and scaling stateful applications with containers the majority of challenges are at the storage layer. Can you expound on this? How are IT teams, software engineers, and database experts overcoming this challenge?

“Well, they are trying to overcome it. It’s a present tense, not a past tense. Storage is currently a major challenge across the enterprise. It seems to be the culprit that’s preventing many innovations from moving forward. New tools like Kubernetes are being introduced to scale and manage storage issues. But the storage issues are massive, and they aren’t going away overnight. Storage complexity isn’t something new, we’re just trying to find novel ways to get around it. We figured we’ll make it smaller with containers and our storage capacity needs will go down. But the fundamental issue is no longer how much storage we need, rather it’s all about where to place specific data and how to share it. Many are finding great success by getting away from quorum drives, which forces us to use SAN, and going back to local storage models that allow us to move data to an archive after it has been read.

Most people haven’t gotten to a point where it’s seamless yet. There’s still a lot of manual intervention that has to happen, even when scaling with containers. Looking at the percentage of the time containers are corrupted, or data is lost show you that the technology hasn’t quite won that battle yet for most users. But we’re a lot closer than we were a year ago.”

Robin: Can you containerize a database? And should you do that before containerizing the application?

“I think that it’s a combination of the two, but people should absolutely start by investigating the database. There are more use cases where containers are not working with databases frankly. But, databases aren’t the only place where data exists. We have data in files, events, or on disks too. When you really come down to it, you have to figure out what your priorities are. If you’re a heavy database user then you have to focus more on that side. Whereas if you’ve gotten out of the databases and are now into say Hadoop clusters, and need to access those files and retrieve data, then you have a slightly different problem. But fundamentally it’s the same challenge regardless of which direction you look.”

Robin: With containers, most people just think about apps. What about the rest of the application lifecycle? Can you containerize everything from database to job scheduling?

“So, I think the answer is the environments have to be hybrid in order for more comprehensive containerization. If containerizing something allows a capability, then you have to try it. It can be the difference between building a competitive edge against other players in your market or falling behind. At this point, winning is all about speed, performance, and agility. The more agile you are, the better off you are. Having that flexibility will provide you true benefit in the long run. I don’t think anyone can rule it out, and I don’t think anyone is being forced into using containers. Right now, we’re at that early stage where testing, development, and proof of concept are happening. It’s certainly not going away. There are true benefits, and you need to figure out what those benefits are for your organization, no matter the stage of the application lifecycle.

Like all technology that’s new and being developed and enhanced, you just need to get into the trenches and try it out. You need to figure out how this technology should work, can work, and will work specifically for your organization. Keep in mind; it will be slightly different than the company next to you.”

Robin: Security has been identified as an issue with containers, does that mean containerizing stateful applications is a better route? What are some opportunities for improving security?

“I read that the very idea of containers came out of security problems and trying to solve them for applications. Containers in and of themselves are in fact highly secure. It’s how they are used that make them challenging from a security perspective. Keep in mind security is a holistic issue. The skillset of your developers also plays a part in that risk. The majority of the time security is handled through the application. How the application produces, consumes, digests, and shares data is all a part of the security picture.

So I would actually look at containers as an improvement upon the security problems of the past, but of course, they are going to introduce new security problems of their own. They are a little different than what we typically find in enterprise applications, so obviously it will take time for security to catch up.”

Robin: What are some of the biggest opportunities to scale container technology in enterprises?

“I have seen great strides with is microservices, and smaller applications or functionality. Outside of that, I am seeing a lot of experimentation at this phase. I frankly don’t see many organizations today going into full-scale production with containers. But, there’s a lot of activity around trying to figure this technology out and utilize it.

Even for organizations that aren’t scaling containers, the majority of them are finding unexpected uses for containers within their enterprises. There are more and more questions coming up about how, why, where, when, and so forth when it comes to bringing that technology into the organization.”

Robin: Do containers offer a solution with the current explosion of data enterprises face?

“Containers absolutely offer a solution. Remember that the whole point of a container is to isolate and reduce complexity. So, you’re looking at much smaller subsets of code and data necessary to run that code. And you’re isolating it from other areas of the application – which is where the performance boost comes in because it doesn’t have to share resources. It has its own place, with its own resources which are dedicated to that space, which is why it’s so efficient. Most database applications that are out there today use shared resources, which is why they are not as robust, and not as flexible and not gaining optimal performance. Containers most definitely offer a solution here.”

Robin: Any additional insights for enterprise teams looking to get into containers?

“I would say containers provide a worthwhile journey. I believe that there are significant benefits across industries. I haven’t found any industry or company that couldn’t benefit from containers. It’s still growing and becoming more innovative. Like a lot of the other big data technologies out there, it can be incorporated into an ecosystem.

Which means it doesn’t have to be looked at as a single solution but as a complement to other solutions you might have in place. There are specific benefits that you get and I think that most organizations should venture into this journey, figure out what those benefits are and then leverage it like any and all technologies that they have. Especially if they are going through a digital transformation and are already geared towards change.”

Craig Brown is a big data architect and data science consultant with more than 25 years experience in the information technology industry. He has a passion and dedication to instructing his clients in both technical and management capabilities that will sustain those demands. His credentials and experience have propelled his career as a leading expert in the IT arena and a top venture capitalist.

Are you looking for advice on scaling stateful applications in your environment with containers? Check out the Robin Systems blog for weekly advice on how to get more out of your entire application lifecycle, powered by containers.


Author Razi Sharir, Vice President Products

More posts by Razi Sharir, Vice President Products