Production Kubernetes Security: Enforcing Zero Trust with Kyverno & OPA Gatekeeper

Scanning and signing can confirm an image is acceptable, but they don't prevent unsafe images from running. Admission control fills this gap. However, many teams either skip this step or only partially implement it, often by just adding a NetworkPolicy and considering the job finished.

This article builds on the last one, where I covered securing the supply chain with Trivy, Cosign, and Falco. I explained that admission control stops harmful workloads from running, not just detecting them after the fact.

In this post, I'll explain how admission control works in Kubernetes, compare Kyverno and Gatekeeper based on real-world use, and share policies to block common risks like root containers, unsigned images, mutable tags, and workloads with unnecessary host access.

How Kubernetes Admission Control Works

Every request to the Kubernetes API goes through authentication, authorisation, and admission before the object is persisted.

• Authentication checks who is making the request. Authorisation, typically handled by RBAC, determines whether a person is permitted to perform the requested action.

• Admission control evaluates the submitted object. Admission plugins and webhooks can allow, modify, or reject the request based on defined policies.

• Many teams overlook this key security step. RBAC might allow someone to create Pods, but it does not verify whether those Pods run as root, use privileged containers, or pull images from untrusted sources. Authorisation answers, “Can you do this?” Admission control answers, “Does this follow our security policies?”

Why Scanning and Signing Alone Aren’t Enough

CI security controls are only effective when deployments follow the expected path. In real production environments, that assumption can break. An engineer may apply a manifest directly with kubectl during an incident, a CI runner could be compromised, or someone might deploy an old manifest that still uses privileged settings.

These situations often cause configuration drift. Admission control helps by adding a checkpoint inside the cluster. It checks workloads as they enter the Kubernetes API, so even if someone skips the usual CI/CD process, admission control can still block risky deployments. CI checks catch problems before deployment, while admission control prevents them from entering the cluster.

Mutating Webhooks vs Validating Webhooks

Kubernetes processes mutating admission first, followed by validating admission.

• Mutating policies can modify a workload by adding labels, setting default values, or injecting a sidecar.

• Validating policies review the workload and then decide to allow or reject it.

For example, Kyverno can automatically add runAsNonRoot: true if it is missing. A validating policy, on the other hand, can block workloads that do not meet this requirement. For important security settings, I prefer validation. It is safer to reject an insecure workload and tell the developer what to fix than to quietly change security-related settings.

Managing Policy as Code with GitOps

Treat security policies as code. Keep them in Git, review changes through pull requests, and use GitOps tools such as Argo CD or Flux to synchronise them with your clusters.

This applies to both namespace-level security settings and Kyverno or Gatekeeper policies. Avoid managing production policies manually with kubectl, because changes can become difficult to review, reproduce, or audit.

Enforcing Pod Security Standards at the Namespace Level

Before adding custom admission policies, Kubernetes already provides Pod Security Admission (PSA) as a built-in baseline. PSA applies Pod Security Standards at the namespace level. For example:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted

This acts as the first layer of enforcement. It sets a basic level of Pod security and doesn't require an extra policy engine. PSA doesn't address every organisation’s unique needs. Tools like Kyverno or Gatekeeper can then add controls for trusted registries, image signatures, resource requirements, and other organisation-specific security requirements.

Extending the Baseline with Kyverno

With the baseline in place, we can add more specific admission policies. For example:

• Blocking Root and Privileged Containers

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-root-and-privileged
spec:
  rules:
    - name: check-runasnonroot
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "Containers must run as non-root and cannot be privileged."
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
            containers:
              - securityContext:
                  privileged: false

• Requiring CPU and Memory Limits on Every Pod

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "CPU and memory requests and limits are required."
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    cpu: "?*"
                    memory: "?*"
                  limits:
                    cpu: "?*"
                    memory: "?*"

• Restricting Deployments to Trusted Registries

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  rules:
    - name: allowed-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "Images must come from myregistry.io."
        pattern:
          spec:
            containers:
              - image: "myregistry.io/*"

• Verifying Cosign Image Signatures at Admission

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  background: false
  rules:
    - name: verify-image-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "myregistry.io/*"
          failureAction: Enforce
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      <your-cosign-public-key>
                      -----END PUBLIC KEY-----

• Blocking ‘latest’ and Other Mutable Image Tags

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  rules:
    - name: require-image-tag
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "Images must use an immutable tag or digest, not 'latest'."
        pattern:
          spec:
            containers:
              - image: "!*: latest"

• Restricting hostPath, hostNetwork, and Privilege Escalation

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-host-access
spec:
  rules:
    - name: disallow-host-namespaces
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "hostPath, hostNetwork, and privilege escalation are not allowed."
        pattern:
          spec:
            =(hostNetwork): false
            =(volumes):
              - X(hostPath): null
            containers:
              - securityContext:
                  allowPrivilegeEscalation: false

Allowing these settings lets a compromised pod break out and access the node directly. Most application workloads do not need this access. Block it by default and allow exceptions only for specific system components that truly need it, with clear documentation.

Audit, Warn, Enforce: The Correct Rollout Order

• Start new policies in Audit mode to identify existing violations without blocking workloads. Review the results, test the policies, and fix legitimate violations before moving to Enforce.

• For Pod Security Admission, Kubernetes also provides Warn mode, which shows violations to users without blocking the request. Avoid switching directly to Enforce in a live cluster. Roll out policies gradually, starting with non-critical workloads, to prevent unexpected deployment failures.

The Full Pipeline: From Pull Request to Running Pod

• Developer opens a pull request; Trivy scans dependencies and secrets.

• On merge, CI builds the image, and Trivy scans it for HIGH and CRITICAL findings.

• CI generates an SBOM, and Cosign signs the image with a key stored in a Secrets Manager.

• The image ships to the registry; ArgoCD or Flux syncs the manifest to the cluster.

• Kyverno or Gatekeeper checks the workload against every policy: signature, registry, tag, resource limits, root and privilege settings, host access.

• If it passes, it's scheduled, and Falco starts watching it; if it fails, it's rejected before a single container starts.

There are four stages in one direction: CI/CD, the registry, admission control in front of the API server, and Falco runtime monitoring. Nothing reaches runtime without passing through the first three.

Real-World Attack Scenarios

These are not just theoretical risks. Recent threat intelligence indicates that attackers are targeting CI/CD pipelines, container registries, and Kubernetes control planes as part of software supply chain attacks. Google Threat Intelligence specifically highlights registry poisoning, unauthorised Kubernetes deployments, privileged workloads, and unsigned or modified container images as relevant attack patterns.

• Direct kubectl Deployment: During an incident, an engineer bypasses the CI/CD pipeline and deploys a workload quickly. If the manifest runs as root or uses excessive privileges, admission policies can reject it before the workload starts. This creates an enforcement boundary even when the normal deployment process is bypassed.

• Repointed latest Tag: An attacker with registry access can replace a legitimate image with a malicious version. Google Threat Intelligence specifically identifies container registry poisoning as a risk when compromised developer or CI/CD credentials overwrite legitimate images. Blocking latest and preferring immutable image references reduces this attack surface.

• Compromised CI Runner: If an attacker takes over a CI/CD workflow or runner, they can use its permissions to upload a changed image or steal credentials. Recent GitHub security advice points out that attackers are targeting CI/CD automation, including workflows and credential theft. By requiring a valid Cosign signature at admission, you add another security check. Even if a bad image gets into the registry, the cluster can block it before it runs.

The key takeaway is that CI/CD security, registry controls, and admission control should not be treated as separate defences. If one layer is compromised, the next layer should still have enough trust information to stop the workload from reaching production.

Handling Exceptions Without Losing Control

Security policies may require exceptions for legitimate workloads, especially system components that require elevated permissions. The goal is to make those exceptions controlled and auditable, not eliminate them.

• Define an owner: Every exception should have a clear owner and reason.

• Manage exceptions as code: Keep Kyverno PolicyException or Gatekeeper exemptions in Git and review them through pull requests.

• Keep them narrow: Scope exceptions only to the workloads that actually need them.

• Review them regularly: Add an expiry or review date so temporary exceptions don't quietly become permanent security gaps.

Common Mistakes with Kyverno and Gatekeeper

• Write small, focused policies instead of trying to cover everything in one. Debugging a rejected deployment with a large, complex policy is hard.

• Remember to use background scanning. Both tools can check resources already running in the cluster, not just new ones. This helps catch drift from before you set up the policy.

• Watch out for policy sprawl. Many overlapping policies with no clear owner can be worse than none, because no one knows what is actually being enforced.

• Do not think of admission control as the final step. It is an important layer, but not the whole security strategy.

Where Kubernetes Policy Enforcement Is Headed

Kubernetes' built-in ValidatingAdmissionPolicy brings simpler policy enforcement directly into the API server. For more advanced needs, Kyverno and Gatekeeper are still strong options. At the same time, Sigstore and SLSA are moving supply-chain security toward stronger provenance, automated verification, and less manual key management.

References

• Kubernetes documentation, kubernetes.io/docs

• CNCF, Cloud Native Security Whitepaper, cncf.io

• Kyverno documentation, kyverno.io

• OPA Gatekeeper documentation, open-policy-agent.github.io/gatekeeper

• Sigstore project, sigstoredev

• SLSA framework, slsadev

• Falco documentation, falco.org

Next in this series: Production Kubernetes Security: Securing Kubernetes Runtime, Network & Workload Identity