The Equifax data breach does not seem to go away – it is clearly an alarming wake-up call for all of us involved in enterprise software. Our most sensitive and private information was exposed in an easily avoidable way. We know code can solve any computer challenge so why is patching still such a pain?
We all assume that the consumer credit reporting agencies employ the absolute best means to protect our data but is there more to it?
The now-former Equifax CEO Richard Smith claims it was a simple human error which tells us that the best means might simply not cut it, there must be a higher-level construct covering data in motion that also accounts for human intervention.
How so, you ask? It is almost like having the best home alarm system and forgetting to turn it on in the right mode or worst yet – not turn it on at all…
The Vulnerable Update Process
Modern technology (microservices architecture included) usually involves countless software components that need to always be properly configured and remain in complete sync not only for security but also performance wise. Updating some or all of the components on a regular basis is fairly common practice as fresh code or updates get tested, verified, and rolled into production.
The update phase where the “older” system and APIs are still running and the “newer” version is being rolled in is a very vulnerable stage. The DevOps paradigm has evolved to address exactly that – trying to automate & accelerate the process by using smaller more granular components at a time while limiting the exposure. While components in the stateless application stack fit this approach, other larger and typically distributed stateful components such as databases require a more elaborate and lengthy process. In fact, the process for patching a database seems to have been the root cause for the breach in the Equifax case.
Lots of vendors offer tools to optimize the key anchors of DevOps – continuous integration (CI) and/or continuous deployment (CD) but only once the new version has been dropped over the wall from Dev to Ops does the fun begin. Often times Ops teams find out (too late…?) that the update/patch does not work as prescribed. The QA arm typically covers the continuous integration part and highlights bugs and issues showcasing the affected functionality. When it comes to deployment, this is a progressive process and each of the many steps along the way might be exposed.
Let’s Avoid Being the Next Equifax
Equifax breach 4 lessons that can be learned
- Completing testing/QA in a non-production environment is mandatory but certainly not sufficient
- Cloning a production environment into a staging/testing environment could prove helpful but requires continuous updates reflecting production changes and is very costly, as well as complex to maintain
- Rolling updates and patches into production should optimally be done “in-place live” and not require partial/incomplete shut-downs of the app/components.
- Rolling updates for an entire business use case (AKA pipeline) requires further checks and balances given the inter-app configuration issues, the sequence of events, scripts, etc.
4 ways to prepare for the future
- Test the entire pipeline in a QA environment configured as close as possible to production
- Break the path or upgrade into smaller parts and trackable procedures enabling smaller impact per event
- Prepare a clear path to roll back from component base to the entire pipeline
- Optimally employ an infrastructure architecture that enables DevOps to run on production
ROBIN Hyper-Converged Kubernetes Platform can help you prepare for the future
At Robin Systems, we heard the news about the Equifax breach and, after freezing our credit, immediately saw that the cause was not a security fault per se but an antiquated process that took a known issue and lengthened the time to put in place a known patch.
In the coming weeks, we will be adding to our stateful expertise with security experts to bring you helpful, in-depth analysis and solutions to help you avoid following in Equifax’s footsteps. Sign up for our mailing list to get the latest delivered to your inbox.