Supply chain security concerns in the build process have been growing since the Solarwinds hack way back in December 2020. That incident compromised Solarwinds’ software build service prior to the signing step, which meant that customers from government agencies, military forces, telcos, and hundreds of others deployed compromised software they assumed was secure. 

Software build systems take the form of a Continuous Integration / Continuous Delivery (CI/CD) system that can automate the build, test and delivery of software artifacts. CI/CD is an agile software development best practice designed to enable more frequent and reliable code updates.

While CI/CD has been widely adopted by software vendors, there are a number of security issues the industry continues to wrestle with, including:

  • Transparency – understanding the original source for all artifacts entering the CI/CD pipeline can improve both security and integrity of built artifacts.
  • Contamination – longer lived processes and/or containers can become polluted, especially if they’re not sealed off from the internet.
  • Reproducibility – consistent output is key to ensuring artifacts are built securely from the same set of inputs – every time.

Supply chain Levels for Software Artifacts (SLSA) is an emerging security framework that can help ensure that all the code you import into your organization and/or build within your organization is done in a secure manner. As such, SLSA is key to helping resolve the transparency and contamination issues, and can put you on the road to achieving reproducibility. 

SLSA Secures the Build Process

SLSA defines a number of “Build Levels” that can provide you with key stages on your road to securing your build service:

Build Level 1: Provenance – any code, library or open source package imported into the organization must have a type of software attestation known as a “provenance attestation” that shows where the code was sourced from and how the package was built:

  • Who built the package (person or system)
  • What process/command was used
  • What the input artifacts (e.g., dependencies) were

A downstream system should be implemented to automatically verify packages were built as expected. 

Build Level 2: Build Service + Signing – introduces a build service that includes signing of the provenance attestation. An isolated signing service ensures against bad actors accessing secrets used to sign the provenance, as well as providing downstream systems/users with signed artifacts to indicate they were not tampered with after being built. 

A downstream service should be implemented to verify the authenticity of the signature.

Build Level 3: Hardened Builds – specifies a number of controls that harden the organization’s build service, including:

  • Build Steps – each build step should have a single responsibility, such as retrieving source code, building an artifact, etc. When that responsibility is fulfilled, the output is checked and, if required, passed to the next step. 
  • Infrastructure – all build steps should have dedicated resources, such as a hardened container in which to execute. Once the step is complete, all those resources should be discarded, preventing contamination of subsequent steps. 
  • Environments – each build step requires an environment in which to run. Care must be taken to ensure the runtime environment is minimized, and the container only includes components that are absolutely required (such as compilers) in order to shrink the attack surface. 
  • Provenance – each build must be dependency complete, and each dependency traceable to the originating source. An out-of-band Software Bill Of Materials (SBOM) should be produced for independent verification. 
  • Service/Network – create a dedicated build service that runs on a segmented network with no internet or manual access in order to limit local exploits and remote tampering/intrusion. This means employing:
    • Pre-Scripted Parameterless Builds – build scripts cannot be accessed and modified within the build service, preventing exploits.
    • Ephemeral, Isolated Build Steps – every step in a build process must execute in its own container, which is discarded at the completion of each step. In other words, containers are purpose-built to perform a single function, reducing the potential for compromise. The goal is to prevent build steps from overstepping their bounds and attempting to exploit common OS services that may be present on the build machine.
    • Hermetically Sealed Environments – containers have no internet access, preventing (for example) dynamic packages from including remote resources. This means all resources, including all open source components must be locally available.

Bonus:

  • Reproducibility – if the same “bits” input don’t always result in the same “bits” output, there’s no guarantee the artifacts you’re working with haven’t changed from build to build. As a best practice, design your build process to fail “safe,” terminating the build if any component generated during a build step fails its checksum verification.

Implementing Declarative Pipelines

In general, CI/CD systems support either a declarative programming model (typically employing YAML), or an imperative programming model based on scripts. Hardening your build service will require a CI/CD solution that allows you to implement a declarative pipeline that breaks down each stage of the pipeline into multiple discrete steps. 

Open Source Declarative Pipeline Tools

The following open source CI/CD solutions can support declarative pipelines:

Jenkins is perhaps the world’s most popular automation server, and offers hundreds of plugins (including a declarative pipeline plugin) to support building, deploying and automating any project on Windows, Linux, and macOS.

Key Capabilities:

  • Highly extensible with a large plugin ecosystem (1500+ plugins) 
  • Integrates with most popular cloud platforms
  • Supports performing work in parallel

Advantage: Best-in-class community support.

Spinnaker (originally from Netflix) is a multi-cloud continuous delivery platform that provides a flexible pipeline management system with integrations to most major cloud providers.

Key Capabilities:

  • Create pipelines that launch and stop server groups, system tests and track rollouts
  • Create immutable images to accelerate rollouts, simplify rollbacks and eliminate configuration drift problems
  • Integrate with monitoring services like Datadog, Prometheus, Stackdriver, or SignalFx for canary analysis

Advantage: Offers a CLI administration tool (Halyard) you can use to install, configure, and upgrade your instances.

Drone offers an Apache 2.0 licensed open source CI platform in which each pipeline step is executed inside an isolated Docker container to ensure security.

Key Capabilities:

  • Integrate with most popular code repos like GitHub, Bitbucket and GitLab.
  • Supports parallel builds and tests
  • Integrates with LambdaTest for cross-browser testing
  • Offers plugins for cloud integration, reporting, testing, notifications, etc

Advantage: By default, isolated Docker containers ensure build steps do not conflict.

Commercial Declarative Pipeline Tools

In addition to the popular open source tools listed above, there are a number of commercially available CI/CD solutions that also support declarative pipelines, including:

TeamCity – created by JetBrains, TeamCity can be installed on Windows and Linux servers. It provides integration with Docker, Visual Studio Team Services, Maven, NuGet, Azure DevOps, Jira Software Cloud, etc, and also supports launching build agents in Kubernetes clusters.

Key Capabilities:

  • Highly extensible/ customizable
  • Runs parallel builds
  • Pipelines are defined using Kotlin-based Domain Specific Language (DSL)

Advantage: Provides for viewing test progress (and history) reports on-the-fly.

CircleCI is available as both a cloud-based and on-premise (self-hosted) solution that supports Windows, Linux, and macOS. It uses a proprietary YAML syntax for its pipelines. Its cloud native CI/CD pipelines are easy to set up as workflows that can be integrated with code repos like GitHub, Bitbucket, etc.

Key Capabilities:

  • Builds can be split and balanced across multiple containers
  • Supports parallel testing 

Advantage: CircleCI Orbs are reusable snippets of code that help automate repetitive tasks and streamline integration with third-party tools.

Travis CI is an early entrant in the CI/CD Security market that initially supported only open-source projects, but subsequently added support for closed-source projects, as well. Travis provides both a cloud-hosted (SaaS) and self-hosted version of their offering that supports more than 30 programming languages on all major OSs, and integrates with common code repos like GitHub and Bitbucket. 

Key Capabilities:

  • Build matrix provides support for parallel builds
  • Offers integration with cloud based testing platform LambdaTest for testing across different browsers, platforms, and devices (emulators)

Advantage: Uses a proprietary YAML syntax that integrates with GitHub Enterprise tools.

There are literally dozens of players in the CI/CD Security market, which is far too many to list here. For an overview of how many of the most popular tools compare, CloudZero offers a comprehensive overview.

Conclusions

Modern software consists mainly (~80%) of open source components. As such, using your hardened build system to build only proprietary code/software releases essentially defeats the purpose of implementing supply chain security. To ensure the security and integrity of your software, you will need to build everything from source code, including all open source components, their complete set of dependencies, and any linked C libraries, as well. 

Unfortunately, creating a SLSA Build Level 3-compliant build service, as well as vendoring all your dependencies in order to build them securely is both time and resource intensive. 

At its core, the ActiveState Platform is a SLSA Build Level 3-compliant, hardened build service that builds all of your open source dependencies (including linked C and Fortran libraries) from source code, and then packages them into a secure runtime environment. ActiveState goes a step further by ensuring all builds are reproducible, as well as generating an SBOM and signed software attestations (both Provenance and Verification Summary Attestations).

By integrating your existing software development process with the ActiveState Platform you can gain SLSA compliance in a matter of days (not months), freeing up your developers to work on what matters most: creating features and functionality.

Next Steps:

Read how ActiveState incorporates SLSA within our platform to ensure the security and integrity of all built open source components.