The Cost Benefits Of Secure Reproducible Environments

Reproducible software environments are not just a good idea – they’re essential to managing risk, driving productivity, and ultimately shipping software faster. All of these can have a direct impact on both the top and bottom line, but most organizations unwittingly continue to muddle along with wikis, manifest files, Docker images/containers, Virtual Machines (VMs), etc., all of which rarely stay up to date with evolving security and innovation needs across development, QA, CI/CD, and production.

 But when all stakeholders can access a single, central source of truth that’s always up to date (such as that provided by ActiveState), organizations can maximize software release throughput by dramatically reducing debugging time, environment configuration drift, and security risks. 

Inconsistent Software Environment Consequences

Think of it this way: inconsistent software environments can have significant costs, both tangible and intangible. If we imagine a large enterprise in the financial services industry with 100 developers across multiple teams and $1B in annual revenue, we can start to understand the financial consequences that reproducible environments can help alleviate. 

Let’s start by investigating the key issues with inconsistent environments, which include:

• Increased Debugging Time – when software behaves differently in development, test, and production environments, developers waste time trying to identify and reproduce bugs only to realize that they are due to environment misconfiguration.
  • Assuming that developers spend an extra 10-20% of their time troubleshooting issues related to environment inconsistencies, and that the loaded cost for a developer is $100K per year, for a large enterprise with 100 developers, we can estimate an annual cost of at least $1M in wasted time.
• Configuration Drift – differences in environment configurations (e.g., versions of dependencies, libraries, OS settings, etc.) can cause applications that work in one environment to fail in another. Most significantly, this can lead to production rollbacks and/or downtime.
  • When deployment failures lead to downtime and/or degraded performance, the result can have a direct impact on revenue generation with losses on the order of thousands or even tens of thousands of dollars per hour (for example, with an ecommerce or financial services company). For our large financial services enterprise, just 5-10 hours of downtime caused by inconsistent environments annually can mean fewer loans, credit card applications, transaction fees, and so on, which might result in a revenue loss of at least $50K per year.
• Increased Security Risks – vulnerabilities patched in one environment may not be uniformly applied across all environments, leaving systems vulnerable to attack. Worse, inconsistent sets of dependencies across environments may mean the organization is unaware which environments are even susceptible to being exploited.
  • Because inconsistent environments lead to an increase in unpatched vulnerabilities, organizations will become more susceptible to security breaches. According to IBM’s 2024 Cost of a Data Breach report, the average cost of a breach is $4.88M based on incident response and compliance costs, as well as reputation damage. 
• Decreased Productivity – developers, QA, DevOps, Ops – in other words, all teams across the Software Development Lifecycle (SDLC) – may be impacted when they find themselves working with different setups, slowing down collaboration and innovation, and wasting time reconfiguring their local environments. But it can also lead to unpredictable performance when trying to scale up the organization’s offering, making it difficult to reliably expand infrastructure and/or the services that run on it. The result is a slow down in development cycles, leading to delayed releases.
  • When time-to-market is delayed, it can cost organizations competitive advantage. Assuming that delays result in missed revenue opportunities and/or losses in market share, it could cost hundreds of thousands or even millions of dollars in lost revenue, depending on the product’s value. For our large enterprise that generates $1B in revenue annually, delays/outages that impact just 1% of those revenues add up to a loss of $10M.

Taken altogether, these issues can ultimately result in customer churn when software bugs, security issues, and performance slowdowns stemming from environment inconsistencies impact customer satisfaction. The 2023 Gartner Report on customer retention estimates that customer churn can cost companies up to 15% of their annual revenue. For our $1B large financial enterprise, if environment inconsistencies contribute to just a 0.5% increase in churn, the cost could be as much as $5M annually.

Summarizing the impact associated with inconsistent software environments for a $1B Financial Services enterprise shows that costs could easily exceed $20M annually, including the costs of recovering from a data breach. 

But by implementing a single, central solution which provides reproducible environments that are always up to date, reflecting changes made by key stakeholders across dev, QA, CI/CD, and production, organizations can avoid losses, generate more revenue, and reduce risk.

ActiveState Reproducible Environments

Reproducibility starts with consistent, reproducible software builds, which leads to consistent, reproducible environments. But reproducible builds, especially for runtime environments, are not the norm. Too many organizations are content to build a runtime environment once – and only once – per project, despite the changes that may be introduced:

• By development teams adding new dependencies in order to innovate or solve an issue.
• By security teams when updating a dependency in order to remediate a critical vulnerability, or else removing it altogether if it’s found to be compromised.
• By compliance teams when removing a dependency upon finding it violates regulatory or licensing terms.

And so on. Innovation needs, compliance requirements, and the need to ensure the security and integrity of the open source software supply chain can all make rebuilds of your runtime environment necessary. But without a reproducible build system in place, your software may be compromised and you wouldn’t even know it. 

A reproducible build is defined as one in which the same bits input always result in the same bits output. To ensure this level of reproducibility, ActiveState has created an automated secure build system based on the Supply chain Levels for Software Artifacts (SLSA) specification. In fact, ActiveState’s secure build service adheres to SLSA Build Level 3, which means:

• Tamper-proof Scripts – build scripts are automated, eliminating a common vector of attack like editing interactive build scripts. 
• Isolated Build Steps – each build step has a single responsibility, such as retrieving source code, building an artifact, etc. When that responsibility is fulfilled, the output is checked and, if required, passed to the next step. 
• Ephemeral Environments – each build step requires an environment, such as a hardened container in which to execute. Once the step is complete all those resources are discarded, preventing contamination of subsequent steps. 
• Provenance – each build step is dependency complete, and each dependency is traceable to its originating source. An out-of-band Provenance Attestation can be used for independent verification. 
• Hermetically Sealed Network – the dedicated build service runs on a locked down system within a segmented network with no internet access, thereby limiting local exploits, as well as remote tampering/intrusion. 

And because builds always start with vetted source code, security risks are minimized. But what happens when a change is made to the resulting runtime environment?

Because the ActiveState Platform provides organizations with a central configuration management console for runtime environment configuration, any stakeholder from development, security, compliance, legal, etc., with the appropriate access rights can make modifications centrally. At that point, ActiveState’s universal build system will automatically rebuild the runtime environment with the following provisions:

• Experimental changes can be made in a branch, and later merged.
• All changes create a git-like snapshot of the environment (along with details about who made which changes, when), and can be automatically reverted, if required. 
• Local environments can be changed without impacting the source of truth on the ActiveState Platform, but users with the appropriate rights can push/pull updates to/from the central server with a single command.

In this way, there’s never any READMEs, wikis, or Docker config files to update, ensuring the single source of truth is always consistent and up to date. With environment reproducibility assured, there are fewer issues slowing down software delivery and deployment, configuration drift is eliminated, and everyone is more productive. 

Conclusions – Environment Reproducibility Reduces Costs

When most organizations think about environment reproducibility, they focus on creating consistent environments across Operating Systems (OSs). While Docker images/containers and VMs can help here, you still need to ensure they’re being built with an up-to-date runtime environment. Without a central way to automate runtime configuration management, organizations spend far too much time and resources manually troubleshooting and reconfiguring environments. And when environments get out of sync, the result is added overhead that slows everyone down, and can result in significant cost penalties.

While assigning a specific dollar value to inconsistent software environments can be challenging because costs vary with organization size, software complexity, and specific industry, the cost analysis provided in this post of $20M annually is highly conservative for a $1B enterprise.  

By standardizing environments, enterprises can mitigate the numerous risks associated with inconsistent environments, streamline development processes, and improve overall system reliability and security.

ActiveState’s universal build system produces secure, reproducible builds that act as the single, central source of truth for all open source runtime environments across development, test and production. That means:

• All developers make use of the same development environment, built according to the organization’s technology standards, internal governance policies, and external regulation requirements.
• Vulnerabilities are centrally remediated, ensuring all environments can be uniformly updated quicker.
• All servers in the CI/CD pipeline pull the same prebuilt runtime environment as that being used by developers, reducing alerts and speeding up CI/CD cycles.
• The production environment is also based on the same runtime environment (usually minus the test harnesses), eliminating the kinds of configuration errors that result in production rollbacks.

And because ActiveState costs only a fraction of the annual cost savings it provides, organizations can realize tangible return on investment, as well as improve productivity, reduce security risk, and comply with standards like the National Institute of Standards and Technologies (NIST) Secure Software Development Framework (SSDF).

Next Steps

Interested in how you can save your organization millions of dollars annually with reproducible software environments? Contact us to learn how our team can help you build a business case for implementing ActiveState’s open source management platform.