Are You Getting the Most from Your Elastic Stack Deployment?

Robin.io Elastic Stack Blog: Part 1

 Note: This blog is Part 1 in a three-part series. If you missed the introduction to the series, you can view it here.

As we mentioned in the kickoff to this blog series, an increasing number of enterprises are using the Elastic Stack for log analysis. It’s a very powerful and useful solution, but unfortunately, most companies aren’t realizing all of the benefits they could from their deployments.

Developers, QA, production, and security teams rely on the Elastic Stack to serve many critical use cases, including log analysis, metrics analysis, application performance monitoring, and security analytics. But in order to meet the needs of these teams, they must be able to quickly provision and easily manage the stack on demand.

In this three-part blog series, we’ll review some of the benefits of running Elastic Stack on virtual machines or bare metal, and then on Kubernetes. After that, we’ll examine some of the most common challenges faced by teams in both environments. The final blog of the series will cover the need for an effective management and automation platform for your Elastic Stack deployment and the functionality you should look for when choosing the right application automation platform for your enterprise.

The Elastic Stack is a collection of open source tools that enables users to ingest data from any source in any format, and search, analyze, and visualize it in real-time. The Elastic Stack consists of three primary components: Elasticsearch, a distributed, JSON-based search and analytics engine; Logstash, a data collection pipeline with native Elasticsearch integration; and Kibana a visualization tool which acts as the user interface for analyzing the data in Elasticsearch. The Stack also has an optional lightweight component called Beats that runs on edge machines in clusters where data is created and ships the data to Logstash and Elasticsearch.

Elastic Stack can be deployed as a hosted, managed service, or can be downloaded and installed on an enterprise’s own hardware or in the cloud. For those who want to provision, manage, and monitor their own deployments from a single console – but prefer not to use a public cloud platform –  it can be installed on-prem using virtual machines (VMs) or on bare metal hardware.

But there are a lot of reasons organizations are not able to unlock the full potential of their Elastic Stack deployments. Let’s examine a few of those challenges:

• The inflexibility caused by creating single, monolithic clusters. When running Elastic Stack, infrastructure teams often lack the bandwidth to cater to custom requests from end users. As a result, they end up creating monolithic clusters, which limits the functionality to a single, common configuration. Giant, shared Elastic clusters can slow down developers and data scientists because they can’t make the necessary customizations for specific versions of Elasticsearch, Logstash, Kibana, or Beats. As a result, developers have to create IT tickets for routine life cycle tasks, including upgrades and scaling, which slows them down even more.
• The excessive effort required to provision custom clusters. The IT department may end up building more than one Elastic cluster to accommodate demand for custom stacks and versions, but the general issues remain. Things may get a little easier if IT has good, pervasive, and up-to-date configuration management for hosts, which helps with scaling out, but the problem of the giant shared Elastic cluster is still a major issue.
• Scaling dynamically to meet sudden demands. When running Elastic, there is no easy way to scale up on-the-fly by adding more memory or CPU if a data node runs out of resources. Scaling out to add more data nodes can also take weeks due to process delays. Since scaling out is very difficult, IT will typically only do it if there’s an ongoing need for a bigger cluster. As a result, the change is either permanent or doesn’t happen at all. Shrinking a cluster is also rare. As a result, Elastic clusters are usually decommissioned as whole units.
• Ensuring adequate security. Without encryption at rest, data is vulnerable. All data should be treated as sensitive and be secured. Storage solutions do not typically provide the necessary security functionality for ensuring data integrity, including encryption at rest for Elastic Stack deployments.
• Multi-cluster strategies can lead to massive hardware costs. Creating dedicated clusters for individual tenants (teams, workloads, and applications) is generally a good strategy. But it requires provisioning additional infrastructure for each cluster for peak capacity, leading to significant hardware underutilization and excessive IT costs.

Stay tuned for Part 2 of our Elastic Stack blog series, where we’ll examine how Kubernetes is solving some ‒ but not all ‒ of the challenges enterprises are facing with their Elastic Stack deployments running on VMs or bare metal hardware.

Learn more in our Elastic Stack Solution Brief.