# Production Kubernetes Security: Building a Zero-Trust Supply Chain with Trivy, Cosign, and Falco

Kubernetes made orchestration easier, but it didn’t fix security. In fact, for many organisations, it quietly made things harder. Teams that once focused on securing a few monolithic servers now manage hundreds of microservices across many namespaces. These are often pulled from public registries, deployed by CI pipelines that rarely receive a full audit, and connected by default unless someone takes steps to restrict them.

This article explains a practical approach to Kubernetes security based on zero-trust principles: verify everything, trust nothing by default, and expect a compromise at any point. We’ll cover the full process, from code commit to running containers in production, using tools now standard for experienced platform teams. These include Trivy for scanning, Cosign and SBOMs for supply chain security, and Falco for runtime detection. The aim is to provide you with a reference architecture, example YAML, and clear reasoning for each choice so you can adapt it to your needs.

### The Kubernetes Threat Landscape:

Most teams don’t realise how broad a Kubernetes cluster's attack surface is until something goes wrong. In general, the risks fall into four main categories.

1.  **Supply chain risk.** Base images pulled from Docker Hub, dependencies pulled from npm or PyPI, and Helm charts pulled from third-party repositories all represent code you didn't write and often didn't review. The 2021 Codecov breach and the more recent XZ Utils backdoor both showed how a single compromised upstream package can propagate into thousands of downstream deployments.
    
2.  **Misconfiguration risk.** Overly permissive RBAC, containers running as root, missing resource limits, and workloads with hostPath mounts remain the most common root causes of Kubernetes incidents.
    
3.  **Runtime risk.** Once a workload is compromised, lateral movement within a cluster is often trivial because pods can communicate with one another and with the Kubernetes API by default. A single vulnerable pod can become a foothold for privilege escalation to node level or cluster admin.
    
4.  **Identity and secrets risk**. Long-lived service account tokens, secrets stored as plain Kubernetes Secrets, and shared credentials across environments turn a contained breach into a full-blown compromise.
    

### Why Traditional Security Fails in Kubernetes:

Traditional security was designed for systems that don't change often. You usually have a firewall, a few servers, and infrastructure that stays the same for weeks or months. Kubernetes is different. Pods are temporary and can be replaced within minutes, often with different IP addresses. Because of this, security based only on IP addresses is no longer enough. Instead, it should be based on workload identity.

Kubernetes deployments also happen much more frequently. With GitOps and CI/CD, applications can be updated several times a day. Manual security reviews cannot keep up, so security checks must be automated throughout the deployment pipeline and at the cluster level. Trust is another challenge. Unlike traditional environments where internal systems were often trusted, every workload in Kubernetes should be verified before it is trusted. This is why Zero Trust has become a core security principle for Kubernetes.

### Layered Defence Architecture: Shift-Left to Runtime.

A mature Kubernetes security setup has five layers, with each one designed to catch what the previous layer might miss.

1.  **Source and dependency scanning** at commit and pull-request time, catching known vulnerabilities and secrets before merging code.
    
2.  **Build-time scanning and signing,** where container images are scanned for vulnerabilities, an SBOM is generated, and the image is cryptographically signed.
    
3.  **Admission control,** where Kubernetes itself refuses to schedule workloads that violate policy, including unsigned images, root containers, or missing resource limits.
    
4.  **Runtime detection,** where tools like Falco monitor process and syscall behaviour for signs of compromise that no static scan could predict.
    
5.  **Continuous verification,** where scheduled scans and drift detection catch newly disclosed vulnerabilities in images that were clean when they were first deployed.
    

![](https://cdn.hashnode.com/uploads/covers/62d3d92a2f40e31decd8c583/7541d481-79a7-4b6b-84a5-407b0bdbb33d.png align="center")

### Supply Chain Security: Trivy, SBOMs, and Cosign

The supply chain layer answers a simple question: can you prove what is actually running in your cluster, and that nobody tampered with it between build and deployment?

*   Trivy scans for vulnerabilities and misconfigurations across container images, filesystems, IaC templates, and even Kubernetes manifests. A typical CI step looks like this:
    

```yaml
- name: Scan image with Trivy
  run: |
    trivy image \
      --severity HIGH,CRITICAL \
      --exit-code 1 \
      --format table \
      myregistry.io/payments-api:${{ github.sha }} 
```

Failing the build on HIGH or CRITICAL findings is deliberate. If a scan runs but never blocks anything, it is a report, not a control.

SBOMs (Software Bills of Materials) provide an auditable inventory of every package, library, and dependency in an image. Generating one is now a single command:

```plaintext
trivy image --format cyclonedx --output sbom.json myregistry.io/payments-api:latest 
```

The real value of an SBOM often appears months later, when a new CVE is found in a library you didn’t even know you used. Instead of re-scanning every image, you can check your stored SBOMs and quickly see which workloads are affected.

Cosign, part of the Sigstore project, solves the tampering problem by signing images cryptographically after they pass scanning.

```plaintext
cosign sign --key cosign.key myregistry.io/payments-api:${GITHUB_SHA}
cosign verify --key cosign.pub myregistry.io/payments-api:${GITHUB_SHA}
```

Together, these three tools help you answer three questions: Is this image vulnerable? What’s inside it? Can I prove it hasn’t been changed since it was built?

### Runtime Threat Detection with Falco:

Admission control prevents bad things from starting, but it can’t help if something goes wrong after a workload is running. That’s where Falco comes in. Falco uses eBPF or a kernel module to watch system calls in real time and checks them against a set of rules. It can catch issues that static scanning misses, because some problems only show up when the process is actually running.

A representative rule for detecting a shell spawned inside a container, a classic sign of a reverse shell or exploited RCE:

```yaml
- rule: Terminal shell in container
  desc: A shell was spawned inside a container
  condition: >
    spawned_process and container and
    shell_procs and proc.tty != 0
  output: >
    Shell spawned in container (user=%user.name container=%container.name
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING 
```

Falco alerts should land somewhere a human can see within minutes, not in a dashboard checked once a week. Wiring Falco to Falcosidekick and routing alerts into Slack, PagerDuty, or a SIEM is the difference between catching an intrusion while it's happening versus finding it in a postmortem three weeks later.

### End-to-End Secure Deployment Pipeline:

Here’s how a typical production pipeline from GitHub or Azure DevOps to Kubernetes usually works:

1.  Developer opens a pull request. Trivy scans the filesystem and dependencies; secrets scanning runs alongside it.
    
2.  On merge, CI builds the container image.
    
3.  Trivy scans the built image for HIGH and CRITICAL vulnerabilities and fails the build if it finds any.
    
4.  It generates an SBOM and stores it alongside the image in an artefact repository.
    
5.  Cosign signs the image using a key held in a Secrets Manager, never committed to the repository.
    
6.  The image is pushed to the registry.
    
7.  ArgoCD or Flux syncs the new manifest to the cluster in line with GitOps practices.
    
8.  Admission control validates the incoming workload for image signatures, non-root user access, resource limits, and the absence of privileged escalation. I'll walk through exactly how in the next article.
    
9.  If the workload passes admission, we'll schedule it. If not, it is rejected before a single container starts.
    
10.  Falco continuously monitors the running workload for anomalous behaviour throughout the pod's lifetime.
     

Each step here is automated and must be followed. Nobody should be able to bypass a signature check by asking nicely in a Slack channel.

### Best Practices and Common Mistakes:

*   Once you’ve tested a policy with real traffic, enforce it in “Enforce” mode, not just “Audit” mode. Teams that leave everything in audit mode are only collecting logs; they're not actively defending their systems.
    
*   Don’t treat SBOMs as just a compliance checkbox. When new CVEs are announced, make sure to actually query them. An unreviewed SBOM is just taking up space.
    
*   Don’t skip the runtime layer just because admission control seems enough. Admission control can’t catch vulnerabilities that are exploited after a signed image is already running.
    

### Future Trends:

Kubernetes security is moving towards more automation and stronger verification.

*   [**Sigstore**](https://www.sigstore.dev/) makes image signing easier by removing the need to manage long-term signing keys.
    
*   [**SLSA**](https://slsa.dev/) helps teams prove software was built securely and in a trusted way.
    
*   [**in-toto**](https://github.com/in-toto/in-toto) verifies every step of the software delivery pipeline, from source code to deployment.
    
*   [**SPIFFE**](https://spiffe.io/) **and SPIRE** give each workload its own secure identity rather than relying on IP addresses.
    

These technologies are the next step in Kubernetes security. The specific tools may change, but the goal stays the same: verify everything, trust nothing by default, and secure every part of the software supply chain.

### **References**

*   Kubernetes documentation, [**kubernetes.io/docs**](http://kubernetes.io/docs)
    
*   CNCF, Cloud Native Security Whitepaper, [**cncf.io**](http://cncf.io)
    
*   Sigstore project, sigstoredev
    
*   SLSA framework, slsadev
    
*   in-toto project, [**in-toto.io**](http://in-toto.io)
    
*   SPIFFE/SPIRE, [**spiffe.io**](http://spiffe.io)
    
*   Trivy documentation, [**aquasecurity.github.io/trivy**](http://aquasecurity.github.io/trivy)
    
*   Kyverno documentation, [**kyverno.io**](http://kyverno.io)
    
*   OPA Gatekeeper documentation, [**open-policy-agent.github.io/gatekeeper**](http://open-policy-agent.github.io/gatekeeper)
    
*   Falco documentation, [**falco.org**](http://falco.org)
    

**Next in this series:** Enforcing Kubernetes Security with Kyverno and OPA Gatekeeper