Gopi Rebala, CTO at OpsMx recently presented a talk at ArgoCon 2023 in Chicago titled “Enforcing Supply Chain Security & Simplifying Compliance Auditing for ArgoCD Deployments”. During the talk, Gopi shared various techniques using frameworks and open source tools like GUAC, Sigstore, in-toto, Elastic, and Grafana to set up a secure supply chain workflow for attestation of entire delivery that integrates into the delivery pipeline.
I’ve summarized the key points from his presentation in this blog. However, if you missed the talk and would like to re-watch it, feel free to watch the entire video below.
Growing threat landscape
Security threats have been on the rise. It is estimated that by 2025, around 45% of companies will become victims to malicious software supply chain attacks conspired by threat actors. One such high-profile attack was the Solarwinds attack of 2020 which is estimated to have infected more than 18,000 systems worldwide, and was affecting numerous users even after 12 months since the attack. as recently as late 2021.
To thwart such attacks on major enterprises and government offices, an Executive Order 14028 was recently issued directing big enterprises to strictly adhere to compliances. Earlier, security compliances were after all a matter of formality, but now the spotlight is on ensuring software supply chain security, especially with ‘security in software delivery’ taking the center stage.
Challenges in Securing Software Delivery
1. Open-source Components in Delivery Pipeline
Most of the components in the software delivery process are open source, and security for open source components is still evolving.
2. Evolving Attack Surface
Entry point for threat actors which was traditionally source code is now evolving to target gaps in CI/CD (integration and delivery processes)
3. Security Data Scattered Across the Ecosystem
Data coming in from tools to ensure security and compliance is scattered across the DevOps ecosystem and only represents a point in time. The data is siloed and not actionable
4. Vulnerability Tracing
Freezing vulnerable code at development doesn’t freeze the constantly changing vulnerability landscape. It becomes challenging to stay updated on the vulnerability status and make changes accordingly
5. Manual Review and Approvals
Manual review and approval steps that make sure developers did what they are supposed to do are time consuming and overwhelming operations teams, reducing overall productivity
6. Manual and Expensive Audits
Answering an audit or compliance request requires a manual review of system logs, which is tedious, time consuming, and impacts productivity
How to ensure end-to-end Security?
There are a number of process changes, tooling adoption and best practices that can be followed by DevOps/ DevSecOps teams to ensure end to end security of their delivery pipeline. Some of them are:
1. Unified DevSecOps Control Plane
- A unified DevSecOps dashboard that can pull data directly from different DevOps tools in the software delivery ecosystem and present a unified view of application deployment status
- A tool that is CD-agnostic and can integrate with different types of tools
2. Continuous Automated Risk Assessment
- Image scanning and CVE assessment on a regular and ongoing basis
- Validating the results from security tools
- Validating dependencies, packages and other 3rd party libraries used
3. Automated Approvals and Security Rules Enforcement
- Evaluate security posture against security and SDLC data
- Automate policy gates to ensure deployments are not blocked for manual approvals
- Manage exceptions
4. Software Delivery Audit and Compliance
- Full deployment audit attestation when regulators come knocking
- Compliance reporting to display security posture
5. Security Vulnerability Tracing / Incident Response
- Continuously monitor application code for new vulnerabilities
- Trace vulnerabilities in current deployments with the help of incident response team
Scalable architecture for securing Software Delivery
Keeping in mind the security challenges that currently pose software delivery, and the priorities that need to be ensured for end-to-end security, here is the 3-step framework that can potentially address the need:
1. Data Collection
This step involves collecting and aggregating data from across the entire DevOps and Security toolchain into one central repository for actionable intelligence.
2. Policy Definition (Policy-as-Code)
Depending on the organization and industry you are operating in, you will have to stay compliant to various policies. By collaborating with different teams such as Dev, Ops, and Security, you need to realize the policies that matter to you, and define those policies as code so they can be automated and acted upon.
3. Policy Enforcement
This is the intelligence layer that must enforce policies defined in step 2 based on the data aggregated from other tools in the delivery ecosystem (step 1). This is the most important step because it brings all the data into one place, blocks/ approves deployment, notifies the concerned personnel responsible, thus automating the entire process.
Now that we’ve understood the architecture of a secure delivery system, let’s understand the framework and tools that can power the process.
Framework & Tools to Secure Software Delivery
1. SLSA framework with GUAC toolchain for securing the build process & Software Composition Analysis:
SLSA (Supply-chain Levels for Software Artifacts) a.k.a “salsa” is a security framework. A checklist of standards and controls to prevent tampering, improve integrity, and secure packages and infrastructure. It’s how you get from “safe enough” to being as resilient as possible, at any link in the chain.
GUAC (Graph for Understanding Artifact Composition) aggregates software security metadata into a high fidelity graph database—normalizing entity identities and mapping standard relationships between them.
2. Sigstore with TUF for secure access of third party components:
Sigstore is an open source project for improving software supply chain security by empowering software developers and consumers to securely sign and verify software artifacts such as release files, container images, binaries, software bills of materials (SBOMs), and more.
The Update Framework (TUF) helps developers maintain the security of software update systems, providing protection even against attackers that compromise the repository or signing keys.
3. DBoM & Policy DB for securing the delivery process:
OpsMx’s Delivery Bill of Materials (DeliveryBOM) captures every step in the application’s software delivery and deployment process – security checks, approvals, policy enforcement, audits, etc. This allows DevSecOps and AppSec teams to demonstrate policy compliance at scale.
4. Policy Controller for securing the deployment process –
Various policy-controller(s) can be used to enforce policy on a Kubernetes cluster based on verifiable supply-chain metadata from cosign
5. Argo Interlace for auditing the deployment process
So how to achieve end-to-end Automation?
Setting up end-to-end security automation is not as easy as it sounds. But the journey we’ve set out on requires the following:
- Sigstore with a trace of all the builds
- Verification in K8s that works with Sigstore to verify the provenance of images
- Ability to do pentesting before deploying to production or to staging (path forward is unclear and the concept is still under work)
- Verifying provenance for build and delivery systems? (still under work as threats could also be present in builds or manifest files)
- Cloud-events or CD-events based on cloud-events that provide standardization across tools to indicate when something goes wrong and allows control to take appropriate action
Parting notes
As an organization, OpsMx is making it easier to aggregate data from the entire DevOps toolchain and expose it as a virtual data layer on top of which predefined policies can be applied and enforced. This will ensure that not only the source code is free from CVEs, but the entire delivery and deployment process is secure against threats.
About OpsMx
OpsMx is a leading innovator and thought leader in the Secure Continuous Delivery space. Leading technology companies such as Google, Cisco, Western Union, among others rely on OpsMx to ship better software faster.
OpsMx Secure CD is the industry’s first CI/CD solution designed for software supply chain security. With built-in compliance controls, automated security assessment, and policy enforcement, OpsMx Secure CD can help you deliver software quickly without sacrificing security.
OpsMx Deploy Shield adds DevSecOps to your existing CI/CD tools with application security orchestration, correlation, and posture management.
0 Comments