Software Supply Chain Security for Cloud Apps

The software supply chain is the same for all modern applications. No matter if you’re building a desktop, mobile, client/server, Web or cloud application, they’re all based on open source software that must be sourced, built and deployed in a secure manner to ensure the integrity and security of your final software product. 

But cloud applications can be more complex than others due to the use of containers (note: while applications can be deployed in the cloud without containers, we’ll focus here on container-based applications). Containers present a number of issues when it comes to supply chain security, including:

1. The need to build secure images
2. The need to build the contents of containers in a secure manner
3. The need to be able to identify the contents of deployed containers in order to understand the potential attack surface
4. The need to monitor containers and their contents over time for security issues
5. The need to update the contents of containers, as required

The bad news is that any one of these steps can introduce security issues into your cloud-based application. The good news is that container-based applications can be updated far more quickly than traditional desktop or server-based applications. Let’s see why. 

Step 1 – Hardened Container Images

Often referred to as “distroless” these images are designed to contain only the runtime dependencies required to run your specific workload, minus the package manager, shells and even most of the operating system. This means smaller images that typically run faster and contain less vulnerabilities. 

Typically based on the Open Container Image (OCI) standard, distroless images can be obtained in a number of ways, including: 

• Ubuntu – Canonical refers to distroless images as “chiseled” (since they’ve been built with their chisel tool) and provides a free set of containers for .Net. Chiseled Ubuntu images are designed to contain the minimal set of dependencies needed to run a production workload, decreasing the attack surface. 
• Wolfi – Chainguard offers prebuilt distroless container images based on the Wolfi Linux distribution that have been built with the goal of minimizing the number of vulnerabilities present. They currently offer hundreds of base images for download. 
• Debian – Google offers a set of Debian-based distroless images that are regularly updated. If none of them meet your requirements, you can also build your own versions using Bazel and then sign them with cosign. 

The caveat with downloading prebuilt images is similar to that for downloading prebuilt open source packages: user beware. If no verifiable proof is provided to ensure the image has been built in a secure manner, you may want to build it yourself.  

Step 2 – Building Secure Containers & Contents

Distroless base images are a great step in the right direction, but when you layer your application on top, your deployable container will only be as secure as your existing software supply chain.  

For this reason, you may want to consider building all of the third-party code – including the Ubuntu/Alpine/Debian packages themselves – from source code in a secure manner. This will require some specialized tooling, such as:

• Bazel – Google’s language agnostic, universal build system that features a declarative pipeline, which allows for every step of the build pipeline to be defined. This means you can add controls at every step to verify the security and integrity of the output.
• Melange – Chainguard’s build system is used to build custom apk (Alpine Package Keeper) packages using declarative pipelines. These are commonly used in conjunction with apko (APK for OCI) in order to create fast, small and reproducible container images.
• Spinnaker – Netflix’s declarative build system is a multi-cloud continuous delivery platform that provides a flexible pipeline management system with integrations to most major cloud providers.

Steps 3/4 – Discoverability & Observability

Identifying what’s been deployed in containers (discoverability), as well as tracking it over time (observability) can be difficult, but it’s essential to ensuring security because as open source components age the more vulnerable they become. For this reason, most Software Composition Analysis (SCA) vendors have extended the capabilities of their tools to include scanning of containers.

While there are a number of commercial SCA vendors with container scanning capabilities, there are also open source tools that do just as good a job at discoverability, including:

• Grype – Anchore’s container scanner will enumerate all open source components and retrieve their vulnerability status. Supports Docker, OCI and Singularity image formats, as well as a wide range of Linux OS’s and open source languages. 
• OSV Scanner – Google’s Open Source Vulnerability (OSV) scanner can be used against containers to provide a list of dependencies and associated CVEs. It obtains its dependency and vulnerability information from the OSV database, which supports a wide range of ecosystems.
• Clair – CoreOS’s open source project for the static analysis of vulnerabilities in OCI and docker containers. The Clair API indexes container images and matches the contents against known vulnerabilities.

Step 5 – Remediability

Containers (not only their contents, but the packages with which the container was built) will inevitably become vulnerable over time. The good news is that, as opposed to other deployment methods, cloud-based containers can be much more quickly updated, especially if your CI/CD system incorporates a decent amount of automated code coverage tests. 

While SCA scanners will flag and notify you when vulnerabilities are found (and some will even automatically generate Jira tickets on your behalf), if you’re not building the contents of your containers from source code you will need to wait for the upstream application to be updated. This can mean a delay of weeks when the time to exploit a new vulnerability can be minutes. 

Building from source provides the capability to update individual components as and when they become vulnerable rather than waiting for an upstream fix. This is the approach that ActiveState takes. We continuously scan images, registries and Kubernetes clusters in order to generate:

• A list of all pods running (regardless of state)
• A manifest of all the open source dependencies in each deployed container
• A dashboard of CVEs that is continually updated, and can be searched and prioritized

When vulnerabilities are found, we can automatically rebuild the containers at scale from source code using the individual updated libraries as/when they become available. Alternatively, you can also choose to make updates one at a time for greater control. See for yourself:

In this way, ActiveState provides a number of advantages over other software supply chain security solutions, including:

• The flexibility to create distroless container images that feature open source software securely built from our catalog of vetted source code, or alternatively added into your own prebuilt container.
• The ability to identify container contents at any point in time, whether they were built by ActiveState or not, and generate a report & dashboard for them. This allows both initial discoverability, as well as ongoing observability in order to decrease Mean Time To Identification (MTTI) of vulnerabilities. 
• The ability to automatically remediate containers on a component by component basis, decreasing Mean Time To Remediation (MTTR).

Next Steps

If hardened, updatable containers sound like something your organization could take advantage of, contact us to discuss how we can help ensure the security of your cloud applications.