Managing the Software Supply Chain

• 1: Licensing
• 2: Lifecycle
• 3: Regulatory Compliance
• 4: Security

1 - Licensing

Software is a form of intellectual property and, in most jurisdictions, the creator of a piece of software is granted various legal rights with respect to the use and replication of this software by others. Typically, the assertion of these rights will be communicated by issuing a software license to consumers of this product.

Any individual or organization violating the terms of a license may be liable to prosecution, hence it is critically important to understand, for every dependency in your solution, that you have the right to legally use that component for the purposes that you intend.

In the case of commercial software, you are required to pay for a license, the price of which may be linked to the number of intended users, the number of instances, or the number of CPUs used to execute it. This payment may be a one-off fee, a recurring charge, or relative to usage. This implies the need to regularly confirm that licenses are still valid.

In the case of Free, Open Source software, the creator has chosen to grant a license without asking a fee. This does not mean that they are not asserting their rights, however, and it is normal for this class of software to come with attached and very specific constraints upon the type of use which is permissible. Again, you must be sure that you have permission to use all of your inherited dependencies in the way that you intend for your product. Some licenses contain ‘copyleft’ restrictions that are viral in nature. These generally grant permission to use, modify and redistribute a component, but only upon the condition that any derived works inherit the same conditions, and/or contribute the source for their modifications back to the community. Again, it is critical to understand the terms granted by every dependency, since you may be inheriting a legal obligation that conflicts with your intended usage or commercial model. Legal precedent now exists to show that this obligation exists at a contractual level when consuming Open Source software and is not merely an expression of copyright.

Some software falls into the class of Public Domain, where the creator has waived, or lost, all their rights, and this software may be used freely without consequence. It is not, however, safe to assume that software distributed without a license is in the public domain.

To manage this challenge safely, it is necessary to maintain a Software Bill of Materials (SBOM) that lists all of the components that make up your product, including all inherited dependencies. For each line item in this SBOM, it is essential to identify the type of license involved, the nature of the terms of this license, and any proof of transaction for commercial products. Whilst it is possible to do this by hand, it is an extremely manually intensive operation that a legal team will usually insist upon prior to every deployment. As such, it is really only viable to use automated scanning as part of your build pipeline. SBOM components that contain machine-readable licenses greatly simplify this process as the alternative is brute-force text scanning to try to identify licenses within each component and parse the license for sensitive terms. This can be expected to result in multiple false positives and negatives, so a necessary approach is to utilize a component library service that maintains a history of component-library relationships that can be manually maintained at an organization-wide level to mitigate the effort involved in tracking this information. This also facilitates the use of policy-based rules for license management.

Now let’s look at the downstream implications. Your product will need to present a license to your customers (and that license may need to inherit clauses from your dependencies). You should provide with your software an easily understood description of the license, the full legal statement of the detail of the license and a machine-readable version of the license such as those maintained by the OSI or Creative Commons. You should also publish the SBOM for your product where your license permits, to simplify recursive dependency management.

If your product is commercial in nature, you will find that licensing can be a costly and resource-intensive overhead. A key element for start-ups in particular is to do as much as possible to automate the on-boarding process for your product, so that your customers can easily integrate your product into their process without manual intervention on your part. This applies equally to paid software modules, or Software-as-a-Service offerings. Simplifying your sales process into something that aligns with your customer’s build pipelines will increase conversion rate and stickiness to an extent that can outweigh a more aggressive, tiered pricing approach requiring a dedicated sales team.

2 - Lifecycle

We often make the mistake of thinking about software development as a linear process with a start and an end: gather some requirements, write some code, ship it, move onto something else. In reality, we are building an asset and the value of that asset is intrinsically linked to its ongoing viability across the lifecycle of the product. We must always think in terms of a circular process of iterative improvement and maintenance as part of our Continuous Delivery process. Software left untouched, starts to rot. The business environment changes, the global marketplace changes, regulations change, new market segments open, so requirements change and our software must be updated in response.

Now consider this in the context of the software supply chain. The same is true of all of your dependencies, and of your customer’s products in turn. Every entry in your SBOM is a product with a lifecycle and your product is an entry in someone else’s SBOM. As a result, as soon as you release a version of your product, it is becoming stale because the dependencies it relies upon are changing. Many of these changes will just represent minor bug fixes or new features that don’t impact you directly, but many will be security fixes for active vulnerabilities, or breaking changes to APIs you rely upon, caused by changing demands upon these products.

First and foremost, you must always ensure that you use explicitly declared versions of every dependency, so that your SBOM remains consistent for a given release. You must also version each component and API in your product so that you are able to introduce change over time without impacting your customers negatively. If you reference a dependency by ’latest’ rather than an explicit version, you may experience random breaking changes in production if those dependencies are dynamically loaded.

Now consider the forces at work across the supply chain. All of your dependencies are either slowly evolving in response to external stimuli, or they have been abandoned and are becoming stale. You must have a strategy and roadmap for updating your system to align to the changes coming from upstream, which involves maintaining your SBOM, enhancing your code to adapt to breaking changes and running your build pipelines regularly to check for unanticipated external factors that break the system.

Beyond this, you must also maintain your own code in response to changing requirements and urgent events such as security issues.

All this has an impact upon your customers. You are free to publish a new release at any time, but your customers may not be able to change their environments at the same pace that you can. In the ideal world, it would be Continuous Delivery all the way down, with everyone updating every few hours in response to changing pressures. In practice, however, there is significant latency across the supply chain and this can lead to tension in the system, and even fracture under some circumstances. If your customers are slower than you to adapt to change then you may need to be able to support multiple versions of your product in production simultaneously, but this quickly increases your overheads and leaves you potentially exposed when change is triggered by security concerns. If you are slower to adapt than your dependencies, you may find that support for key components is withdrawn before you are ready and you end up in a state where you can no longer build your product without major remedial actions.

It is essential to recognize this pattern of waves of change that ripple up and down the supply chain and invest in your Continuous Delivery process to mitigate the impact upon your part of the process.

3 - Regulatory Compliance

In many cases, you may be operating in an environment that requires that you conform to predefined standards of compliance to corporate and/or government regulation.

In a pattern similar to that discussed in licensing, you will therefore have to inspect all inherited dependencies in your SBOM to confirm that you are compliant. An example of this would be the International Traffic in Arms regulations (ITAR) which restrict and control the export of defense and military related technologies and covers certain classes of software asset that may not be exported to some countries. In a similar manner, it is therefore necessary to include a step in the build pipeline that scans for non-compliant articles. This usually requires a combination of text searches and centrally managed, policy-driven lookup tables.

Compliance always implies external audit, so your pipeline steps must produce audited output that can be preserved for the lifetime of the product and presented to internal and external audiences for validation. It should be assumed that this data may at any point become required as part of a legal defense, so forensic cleanliness should be observed at all times, generating immutable records with appropriate levels of tamper-proofing and non-repudiation built in.

4 - Security

Any software assets that your product depends upon can potentially be used to introduce vulnerabilities into your product, or to spread those vulnerabilities to your customers. It is critical therefore, to take appropriate action to mitigate this risk, otherwise you may potentially be held liable for damages caused to your customers as a result of a subsequent breach, or incur irreparable reputational damage to your product or business.

• You must be able to identify all software assets that your product depends upon, and be able to establish the ongoing provenance of each of these assets.

• You must have systems in place to allow you to identify and immediately act upon tampering attempts within your software supply chain.

• You must be able to secure the infrastructure that you use to create your product, detect and act upon attempts to subvert your build process by attackers.

• You should be able to provide strong assurances of the integrity of your build process as provenance to your customers.

The following are examples of potential attack vectors which must be secured against:

Individuals may attempt to deliberately introduce vulnerabilities directly into your source code. In the Open Source space, this may be in the form of adversarial pull requests containing malicious code hidden within feature submissions or bug fixes. In commercial software, this may take the form of insider attacks or revenge from disgruntled employees. Multiple peer reviews of code submissions should be used to mitigate this risk, with careful consideration for social engineering that undermines this people-process, such as sock-puppet review accounts.

Attackers may seek to inject code into your source by compromising your source code repository. This may be in the form of attacks upon outsourced service providers, permitting access to your codebase, or supply chain attacks upon the integrity of the software assets used to implement self-hosted source code repositories. You should be auditing all changes to the codebase and correlating these against known work-stream activities in order to identify and act upon unexpected changes.

By subverting your build infrastructure, it is possible for attackers to substitute alternate source code during your build process. Mitigating against this requires a chain of provenance that can be established independent from an individual build step to enable tampering to be detected. The security of your build process should receive similar investment to the security of your end product as one depends entirely upon the other.

A bait-and-switch attack can involve the introduction of an initially innocuous dependency which is then later replaced with a malicious payload, upstream of your build process. Under these circumstances, a one-off code review and dependency evaluation is insufficient to detect the attack. Instead, it is necessary to validate the provenance of the dependency during each build so that the substitution may be found as soon as it occurs.

Similarly, it can be possible to substitute assets that live in image repositories, post-build, but which are subsequently utilized for deployments. Thus, the item deployed is not the one built and tested. Mitigation requires the extension of your provenance system into your production deployment process, or to your customer’s build and deployment process. The same attack can apply to third party mirror sites for common dependencies used dynamically during deployment.

Denial of service attacks can be employed by attackers by removing key dependencies from your upstream supply chain, or overloading network resources that provide access to these components. This risk can be mitigated by caching copies of these assets locally, but be aware that this brings with it all of the challenges associated with maintaining the validity of cached data, which can include other security vulnerabilities such as failure to detect and react in a timely manner to new vulnerabilities reported against existing, cached dependencies (zero-day CVEs etc.), or the use of cache-poisoning to inject vulnerabilities.

Static and dynamic security analysis tools can be used as part of the build process but it is important to recognize that these also form part of your software supply chain dependencies and are themselves also potentially vulnerable to attack. Meta-analysis currently shows that no single security analysis tool on the market provides comprehensive identification of all currently known vulnerabilities, so it is essential to adopt a defense-in-depth strategy, leveraging multiple, parallel solutions to maximize your chances of early identification of problems.

Be aware that corporate and state-level firewall solutions may deliberately substitute or inject changes into assets which are downloaded through these routes and subsequently consumed by build processes. The same may apply to the software assets that you supply to your customers.

It is particularly important to consider the security of infrastructure used to train ML models, since it is now practical to inject vulnerabilities into models which are mathematically hard to detect with analytical testing but which can be remotely enabled through the use of specific data injected as input to production systems through the front door.

Appropriate implementation of security best practice usually requires some degree of separation of concerns. If we consider the three aspects of ‘maintaining the product codebase’, ‘maintaining the build and release governance process’ and ‘maintaining the audit trail’, it is clear that anyone who has access to all three is empowered to subvert the process without detection. As such, it is advisable to limit the number of people who are required to have full visibility and access to all of these aspects of your continuous delivery process, and to ensure that additional audit processes exist to monitor changes that span all areas.

Consideration should be given to the SLSA specification for supply chain security which may be found at slsa.dev.