<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[The DevOps Space]]></title><description><![CDATA[The DevOps Space — practical insights, tutorials, and real-world practices across DevOps, cloud, Kubernetes, automation, security, and observability.]]></description><link>https://blog.muhammadrafay.com</link><image><url>https://cdn.hashnode.com/uploads/logos/62d3d92a2f40e31decd8c583/f0924c24-13ba-4a06-88d4-035068478f91.png</url><title>The DevOps Space</title><link>https://blog.muhammadrafay.com</link></image><generator>RSS for Node</generator><lastBuildDate>Tue, 01 Sep 2026 13:48:56 GMT</lastBuildDate><atom:link href="https://blog.muhammadrafay.com/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Production Kubernetes Security: Enforcing Zero Trust with Kyverno & OPA Gatekeeper]]></title><description><![CDATA[Scanning and signing can confirm an image is acceptable, but they don't prevent unsafe images from running. Admission control fills this gap. However, many teams either skip this step or only partiall]]></description><link>https://blog.muhammadrafay.com/kubernetes-security-zero-trust-kyverno-opa-gatekeeper</link><guid isPermaLink="true">https://blog.muhammadrafay.com/kubernetes-security-zero-trust-kyverno-opa-gatekeeper</guid><category><![CDATA[Kubernetes]]></category><category><![CDATA[Kubernetes Security]]></category><category><![CDATA[DevSecOps]]></category><category><![CDATA[kyverno]]></category><category><![CDATA[OPA Gatekeeper]]></category><category><![CDATA[cloud native security]]></category><category><![CDATA[zero-trust]]></category><dc:creator><![CDATA[Muhammad Rafay]]></dc:creator><pubDate>Sat, 15 Aug 2026 13:48:34 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/62d3d92a2f40e31decd8c583/f480b0a0-8271-4d8c-8987-f72610c3abfd.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Scanning and signing can confirm an image is acceptable, but they don't prevent unsafe images from running. <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/"><strong>Admission control</strong></a> fills this gap. However, many teams either skip this step or only partially implement it, often by just adding a NetworkPolicy and considering the job finished.</p>
<p>This article builds on the last one, where I covered securing the <a href="https://www.linkedin.com/pulse/production-kubernetes-security-building-zero-trust-supply-rafay-d0wrf/"><strong>supply chain with Trivy, Cosign, and Falco.</strong></a> I explained that admission control stops harmful workloads from running, not just detecting them after the fact.</p>
<p>In this post, I'll explain how admission control works in Kubernetes, compare <a href="https://kyverno.io/"><strong>Kyverno</strong></a> and <a href="https://kubernetes.io/blog/2019/08/06/opa-gatekeeper-policy-and-governance-for-kubernetes/"><strong>Gatekeeper</strong></a> based on real-world use, and share policies to block common risks like <strong>root containers</strong>, <strong>unsigned images</strong>, <strong>mutable tags</strong>, and <strong>workloads</strong> with unnecessary host access.</p>
<h3><strong>How Kubernetes Admission Control Works</strong></h3>
<p>Every request to the Kubernetes API goes through <strong>authentication, authorisation, and admission</strong> before the object is persisted.</p>
<ul>
<li><p><strong>Authentication</strong> checks who is making the request. <strong>Authorisation</strong>, typically handled by RBAC, determines whether a person is permitted to perform the requested action.</p>
</li>
<li><p>Admission control evaluates the submitted object. Admission plugins and webhooks can allow, modify, or reject the request based on defined policies.</p>
</li>
<li><p>Many teams overlook this key security step. RBAC might allow someone to create Pods, but it does not verify whether those Pods run as root, use privileged containers, or pull images from untrusted sources. Authorisation answers, <strong>“Can you do this?”</strong> Admission control answers, <strong>“Does this follow our security policies?”</strong></p>
</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/62d3d92a2f40e31decd8c583/f5849b2d-f90f-452b-a06b-7eaa6efcf1c4.png" alt="" style="display:block;margin:0 auto" />

<h3><strong>Why Scanning and Signing Alone Aren’t Enough</strong></h3>
<p>CI security controls are only effective when deployments follow the expected path. In real production environments, that assumption can break. An engineer may apply a manifest directly with kubectl during an incident, a CI runner could be compromised, or someone might deploy an old manifest that still uses privileged settings.</p>
<p>These situations often cause configuration drift. Admission control helps by adding a checkpoint inside the cluster. It checks workloads as they enter the Kubernetes API, so even if someone skips the usual CI/CD process, admission control can still block risky deployments. CI checks catch problems before deployment, while admission control prevents them from entering the cluster.</p>
<h3><strong>Mutating Webhooks vs Validating Webhooks</strong></h3>
<p>Kubernetes processes <strong>mutating admission first</strong>, followed by <strong>validating admission</strong>.</p>
<ul>
<li><p><strong>Mutating</strong> policies can modify a workload by adding labels, setting default values, or injecting a sidecar.</p>
</li>
<li><p><strong>Validating</strong> policies review the workload and then decide to allow or reject it.</p>
</li>
</ul>
<p><strong>For example,</strong> Kyverno can automatically add <strong>runAsNonRoot: true</strong> if it is missing. A validating policy, on the other hand, can block workloads that do not meet this requirement. For important security settings, I prefer validation. It is safer to reject an insecure workload and tell the developer what to fix than to quietly change security-related settings.</p>
<h3><strong>Managing Policy as Code with GitOps</strong></h3>
<p>Treat security policies as code. Keep them in Git, review changes through pull requests, and use GitOps tools such as Argo CD or Flux to synchronise them with your clusters.</p>
<p>This applies to both <strong>namespace-level security settings</strong> and <strong>Kyverno or Gatekeeper policies</strong>. Avoid managing production policies manually with kubectl, because changes can become difficult to review, reproduce, or audit.</p>
<h3><strong>Enforcing Pod Security Standards at the Namespace Level</strong></h3>
<p>Before adding custom admission policies, Kubernetes already provides <a href="https://kubernetes.io/docs/concepts/security/pod-security-admission/"><strong>Pod Security Admission (PSA)</strong></a> as a built-in baseline. PSA applies Pod Security Standards at the <strong>namespace level</strong>. For example:</p>
<pre><code class="language-yaml">apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
</code></pre>
<p>This acts as the <strong>first layer of enforcement</strong>. It sets a basic level of Pod security and doesn't require an extra policy engine. PSA doesn't address every organisation’s unique needs. Tools like Kyverno or Gatekeeper can then add controls for <strong>trusted registries, image signatures, resource requirements, and other organisation-specific security requirements.</strong></p>
<h3><strong>Extending the Baseline with Kyverno</strong></h3>
<p>With the baseline in place, we can add more specific admission policies. For example:</p>
<ul>
<li><strong>Blocking Root and Privileged Containers</strong></li>
</ul>
<pre><code class="language-yaml">apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-root-and-privileged
spec:
  rules:
    - name: check-runasnonroot
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "Containers must run as non-root and cannot be privileged."
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
            containers:
              - securityContext:
                  privileged: false
</code></pre>
<ul>
<li><strong>Requiring CPU and Memory Limits on Every Pod</strong></li>
</ul>
<pre><code class="language-yaml">apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "CPU and memory requests and limits are required."
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    cpu: "?*"
                    memory: "?*"
                  limits:
                    cpu: "?*"
                    memory: "?*"
</code></pre>
<ul>
<li><strong>Restricting Deployments to Trusted Registries</strong></li>
</ul>
<pre><code class="language-yaml">apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  rules:
    - name: allowed-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "Images must come from myregistry.io."
        pattern:
          spec:
            containers:
              - image: "myregistry.io/*"
</code></pre>
<ul>
<li><strong>Verifying Cosign Image Signatures at Admission</strong></li>
</ul>
<pre><code class="language-yaml">apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  background: false
  rules:
    - name: verify-image-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "myregistry.io/*"
          failureAction: Enforce
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      &lt;your-cosign-public-key&gt;
                      -----END PUBLIC KEY-----
</code></pre>
<ul>
<li><strong>Blocking ‘latest’ and Other Mutable Image Tags</strong></li>
</ul>
<pre><code class="language-yaml">apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  rules:
    - name: require-image-tag
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "Images must use an immutable tag or digest, not 'latest'."
        pattern:
          spec:
            containers:
              - image: "!*: latest"
</code></pre>
<ul>
<li><strong>Restricting hostPath, hostNetwork, and Privilege Escalation</strong></li>
</ul>
<pre><code class="language-yaml">apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-host-access
spec:
  rules:
    - name: disallow-host-namespaces
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        failureAction: Enforce
        message: "hostPath, hostNetwork, and privilege escalation are not allowed."
        pattern:
          spec:
            =(hostNetwork): false
            =(volumes):
              - X(hostPath): null
            containers:
              - securityContext:
                  allowPrivilegeEscalation: false
</code></pre>
<p>Allowing these settings lets a compromised pod break out and access the node directly. Most application workloads do not need this access. Block it by default and allow exceptions only for specific system components that truly need it, with clear documentation.</p>
<h3><strong>Audit, Warn, Enforce: The Correct Rollout Order</strong></h3>
<ul>
<li><p>Start new policies in <strong>Audit</strong> mode to identify existing violations without blocking workloads. Review the results, test the policies, and fix legitimate violations before moving to <strong>Enforce</strong>.</p>
</li>
<li><p>For <strong>Pod Security Admission</strong>, Kubernetes also provides <strong>Warn</strong> mode, which shows violations to users without blocking the request. Avoid switching directly to <strong>Enforce</strong> in a live cluster. Roll out policies gradually, starting with non-critical workloads, to prevent unexpected deployment failures.</p>
</li>
</ul>
<h3><strong>The Full Pipeline: From Pull Request to Running Pod</strong></h3>
<ul>
<li><p>Developer opens a pull request; Trivy scans dependencies and secrets.</p>
</li>
<li><p>On merge, CI builds the image, and Trivy scans it for HIGH and CRITICAL findings.</p>
</li>
<li><p>CI generates an SBOM, and Cosign signs the image with a key stored in a Secrets Manager.</p>
</li>
<li><p>The image ships to the registry; ArgoCD or Flux syncs the manifest to the cluster.</p>
</li>
<li><p>Kyverno or Gatekeeper checks the workload against every policy: signature, registry, tag, resource limits, root and privilege settings, host access.</p>
</li>
<li><p>If it passes, it's scheduled, and Falco starts watching it; if it fails, it's rejected before a single container starts.</p>
</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/62d3d92a2f40e31decd8c583/c3be6997-8518-4785-8d45-68b03ea95d72.png" alt="" style="display:block;margin:0 auto" />

<p><strong>There are four stages in one direction:</strong> CI/CD, the registry, admission control in front of the API server, and Falco runtime monitoring. Nothing reaches runtime without passing through the first three.</p>
<h3><strong>Real-World Attack Scenarios</strong></h3>
<p>These are not just theoretical risks. Recent threat intelligence indicates that attackers are targeting <strong>CI/CD pipelines, container registries, and Kubernetes control planes</strong> as part of software supply chain attacks. <a href="https://cloud.google.com/blog/topics/threat-intelligence/preparation-hardening-destructive-attacks?utm_source=chatgpt.com"><strong>Google Threat Intelligence</strong></a> specifically highlights registry poisoning, unauthorised Kubernetes deployments, privileged workloads, and unsigned or modified container images as relevant attack patterns.</p>
<ul>
<li><p><strong>Direct kubectl Deployment:</strong> During an incident, an engineer bypasses the CI/CD pipeline and deploys a workload quickly. If the manifest runs as root or uses excessive privileges, admission policies can reject it before the workload starts. This creates an enforcement boundary even when the normal deployment process is bypassed.</p>
</li>
<li><p><strong>Repointed latest Tag:</strong> An attacker with registry access can replace a legitimate image with a malicious version. Google Threat Intelligence specifically identifies <strong>container registry poisoning</strong> as a risk when compromised developer or CI/CD credentials overwrite legitimate images. Blocking latest and preferring immutable image references reduces this attack surface.</p>
</li>
<li><p><strong>Compromised CI Runner:</strong> If an attacker takes over a CI/CD workflow or runner, they can use its permissions to upload a changed image or steal credentials. Recent <a href="https://github.blog/changelog/2026-07-28-github-actions-holds-potentially-malicious-workflows-for-approval/?utm_source=chatgpt.com"><strong>GitHub security advice points</strong></a> out that attackers are targeting CI/CD automation, including workflows and credential theft. By requiring a valid Cosign signature at admission, you add another security check. Even if a bad image gets into the registry, the cluster can block it before it runs.</p>
</li>
</ul>
<p><strong>The key takeaway is that</strong> CI/CD security, registry controls, and admission control should not be treated as separate defences. If one layer is compromised, the next layer should still have enough trust information to stop the workload from reaching production.</p>
<h3><strong>Handling Exceptions Without Losing Control</strong></h3>
<p>Security policies may require exceptions for legitimate workloads, especially system components that require elevated permissions. The goal is to make those exceptions <strong>controlled and auditable</strong>, not eliminate them.</p>
<ul>
<li><p><strong>Define an owner:</strong> Every exception should have a clear owner and reason.</p>
</li>
<li><p><strong>Manage exceptions as code:</strong> Keep Kyverno PolicyException or Gatekeeper exemptions in Git and review them through pull requests.</p>
</li>
<li><p><strong>Keep them narrow:</strong> Scope exceptions only to the workloads that actually need them.</p>
</li>
<li><p><strong>Review them regularly:</strong> Add an expiry or review date so temporary exceptions don't quietly become permanent security gaps.</p>
</li>
</ul>
<h3><strong>Common Mistakes with Kyverno and Gatekeeper</strong></h3>
<ul>
<li><p>Write small, focused policies instead of trying to cover everything in one. Debugging a rejected deployment with a large, complex policy is hard.</p>
</li>
<li><p>Remember to use background scanning. Both tools can check resources already running in the cluster, not just new ones. This helps catch drift from before you set up the policy.</p>
</li>
<li><p>Watch out for policy sprawl. Many overlapping policies with no clear owner can be worse than none, because no one knows what is actually being enforced.</p>
</li>
<li><p>Do not think of admission control as the final step. It is an important layer, but not the whole security strategy.</p>
</li>
</ul>
<h3><strong>Where Kubernetes Policy Enforcement Is Headed</strong></h3>
<p>Kubernetes' built-in ValidatingAdmissionPolicy brings simpler policy enforcement directly into the API server. For more advanced needs, Kyverno and Gatekeeper are still strong options. At the same time, Sigstore and SLSA are moving supply-chain security toward stronger provenance, automated verification, and less manual key management.</p>
<h3><strong>References</strong></h3>
<ul>
<li><p>Kubernetes documentation, <a href="http://kubernetes.io/docs"><strong>kubernetes.io/docs</strong></a></p>
</li>
<li><p>CNCF, Cloud Native Security Whitepaper, <a href="http://cncf.io"><strong>cncf.io</strong></a></p>
</li>
<li><p>Kyverno documentation, <a href="http://kyverno.io"><strong>kyverno.io</strong></a></p>
</li>
<li><p>OPA Gatekeeper documentation, <a href="http://open-policy-agent.github.io/gatekeeper"><strong>open-policy-agent.github.io/gatekeeper</strong></a></p>
</li>
<li><p>Sigstore project, sigstoredev</p>
</li>
<li><p>SLSA framework, slsadev</p>
</li>
<li><p>Falco documentation, <a href="http://falco.org"><strong>falco.org</strong></a></p>
</li>
</ul>
<p><strong>Next in this series:</strong> Production Kubernetes Security: Securing Kubernetes Runtime, Network &amp; Workload Identity</p>
]]></content:encoded></item><item><title><![CDATA[Production Kubernetes Security: Building a Zero-Trust Supply Chain with Trivy, Cosign, and Falco]]></title><description><![CDATA[Kubernetes made orchestration easier, but it didn’t fix security. In fact, for many organisations, it quietly made things harder. Teams that once focused on securing a few monolithic servers now manag]]></description><link>https://blog.muhammadrafay.com/kubernetes-security-zero-trust-supply-chain</link><guid isPermaLink="true">https://blog.muhammadrafay.com/kubernetes-security-zero-trust-supply-chain</guid><dc:creator><![CDATA[Muhammad Rafay]]></dc:creator><pubDate>Sat, 15 Aug 2026 11:49:07 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/62d3d92a2f40e31decd8c583/8d88e6bd-8eae-443f-a4fb-fac247116938.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Kubernetes made orchestration easier, but it didn’t fix security. In fact, for many organisations, it quietly made things harder. Teams that once focused on securing a few monolithic servers now manage hundreds of microservices across many namespaces. These are often pulled from public registries, deployed by CI pipelines that rarely receive a full audit, and connected by default unless someone takes steps to restrict them.</p>
<p>This article explains a practical approach to Kubernetes security based on zero-trust principles: verify everything, trust nothing by default, and expect a compromise at any point. We’ll cover the full process, from code commit to running containers in production, using tools now standard for experienced platform teams. These include Trivy for scanning, Cosign and SBOMs for supply chain security, and Falco for runtime detection. The aim is to provide you with a reference architecture, example YAML, and clear reasoning for each choice so you can adapt it to your needs.</p>
<h3>The Kubernetes Threat Landscape:</h3>
<p>Most teams don’t realise how broad a Kubernetes cluster's attack surface is until something goes wrong. In general, the risks fall into four main categories.</p>
<ol>
<li><p><strong>Supply chain risk.</strong> Base images pulled from Docker Hub, dependencies pulled from npm or PyPI, and Helm charts pulled from third-party repositories all represent code you didn't write and often didn't review. The 2021 Codecov breach and the more recent XZ Utils backdoor both showed how a single compromised upstream package can propagate into thousands of downstream deployments.</p>
</li>
<li><p><strong>Misconfiguration risk.</strong> Overly permissive RBAC, containers running as root, missing resource limits, and workloads with hostPath mounts remain the most common root causes of Kubernetes incidents.</p>
</li>
<li><p><strong>Runtime risk.</strong> Once a workload is compromised, lateral movement within a cluster is often trivial because pods can communicate with one another and with the Kubernetes API by default. A single vulnerable pod can become a foothold for privilege escalation to node level or cluster admin.</p>
</li>
<li><p><strong>Identity and secrets risk</strong>. Long-lived service account tokens, secrets stored as plain Kubernetes Secrets, and shared credentials across environments turn a contained breach into a full-blown compromise.</p>
</li>
</ol>
<h3>Why Traditional Security Fails in Kubernetes:</h3>
<p>Traditional security was designed for systems that don't change often. You usually have a firewall, a few servers, and infrastructure that stays the same for weeks or months. Kubernetes is different. Pods are temporary and can be replaced within minutes, often with different IP addresses. Because of this, security based only on IP addresses is no longer enough. Instead, it should be based on workload identity.</p>
<p>Kubernetes deployments also happen much more frequently. With GitOps and CI/CD, applications can be updated several times a day. Manual security reviews cannot keep up, so security checks must be automated throughout the deployment pipeline and at the cluster level. Trust is another challenge. Unlike traditional environments where internal systems were often trusted, every workload in Kubernetes should be verified before it is trusted. This is why Zero Trust has become a core security principle for Kubernetes.</p>
<h3>Layered Defence Architecture: Shift-Left to Runtime.</h3>
<p>A mature Kubernetes security setup has five layers, with each one designed to catch what the previous layer might miss.</p>
<ol>
<li><p><strong>Source and dependency scanning</strong> at commit and pull-request time, catching known vulnerabilities and secrets before merging code.</p>
</li>
<li><p><strong>Build-time scanning and signing,</strong> where container images are scanned for vulnerabilities, an SBOM is generated, and the image is cryptographically signed.</p>
</li>
<li><p><strong>Admission control,</strong> where Kubernetes itself refuses to schedule workloads that violate policy, including unsigned images, root containers, or missing resource limits.</p>
</li>
<li><p><strong>Runtime detection,</strong> where tools like Falco monitor process and syscall behaviour for signs of compromise that no static scan could predict.</p>
</li>
<li><p><strong>Continuous verification,</strong> where scheduled scans and drift detection catch newly disclosed vulnerabilities in images that were clean when they were first deployed.</p>
</li>
</ol>
<img src="https://cdn.hashnode.com/uploads/covers/62d3d92a2f40e31decd8c583/7541d481-79a7-4b6b-84a5-407b0bdbb33d.png" alt="" style="display:block;margin:0 auto" />

<h3>Supply Chain Security: Trivy, SBOMs, and Cosign</h3>
<p>The supply chain layer answers a simple question: can you prove what is actually running in your cluster, and that nobody tampered with it between build and deployment?</p>
<ul>
<li>Trivy scans for vulnerabilities and misconfigurations across container images, filesystems, IaC templates, and even Kubernetes manifests. A typical CI step looks like this:</li>
</ul>
<pre><code class="language-yaml">- name: Scan image with Trivy
  run: |
    trivy image \
      --severity HIGH,CRITICAL \
      --exit-code 1 \
      --format table \
      myregistry.io/payments-api:${{ github.sha }} 
</code></pre>
<p>Failing the build on HIGH or CRITICAL findings is deliberate. If a scan runs but never blocks anything, it is a report, not a control.</p>
<p>SBOMs (Software Bills of Materials) provide an auditable inventory of every package, library, and dependency in an image. Generating one is now a single command:</p>
<pre><code class="language-plaintext">trivy image --format cyclonedx --output sbom.json myregistry.io/payments-api:latest 
</code></pre>
<p>The real value of an SBOM often appears months later, when a new CVE is found in a library you didn’t even know you used. Instead of re-scanning every image, you can check your stored SBOMs and quickly see which workloads are affected.</p>
<p>Cosign, part of the Sigstore project, solves the tampering problem by signing images cryptographically after they pass scanning.</p>
<pre><code class="language-plaintext">cosign sign --key cosign.key myregistry.io/payments-api:${GITHUB_SHA}
cosign verify --key cosign.pub myregistry.io/payments-api:${GITHUB_SHA}
</code></pre>
<p>Together, these three tools help you answer three questions: Is this image vulnerable? What’s inside it? Can I prove it hasn’t been changed since it was built?</p>
<h3>Runtime Threat Detection with Falco:</h3>
<p>Admission control prevents bad things from starting, but it can’t help if something goes wrong after a workload is running. That’s where Falco comes in. Falco uses eBPF or a kernel module to watch system calls in real time and checks them against a set of rules. It can catch issues that static scanning misses, because some problems only show up when the process is actually running.</p>
<p>A representative rule for detecting a shell spawned inside a container, a classic sign of a reverse shell or exploited RCE:</p>
<pre><code class="language-yaml">- rule: Terminal shell in container
  desc: A shell was spawned inside a container
  condition: &gt;
    spawned_process and container and
    shell_procs and proc.tty != 0
  output: &gt;
    Shell spawned in container (user=%user.name container=%container.name
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING 
</code></pre>
<p>Falco alerts should land somewhere a human can see within minutes, not in a dashboard checked once a week. Wiring Falco to Falcosidekick and routing alerts into Slack, PagerDuty, or a SIEM is the difference between catching an intrusion while it's happening versus finding it in a postmortem three weeks later.</p>
<h3>End-to-End Secure Deployment Pipeline:</h3>
<p>Here’s how a typical production pipeline from GitHub or Azure DevOps to Kubernetes usually works:</p>
<ol>
<li><p>Developer opens a pull request. Trivy scans the filesystem and dependencies; secrets scanning runs alongside it.</p>
</li>
<li><p>On merge, CI builds the container image.</p>
</li>
<li><p>Trivy scans the built image for HIGH and CRITICAL vulnerabilities and fails the build if it finds any.</p>
</li>
<li><p>It generates an SBOM and stores it alongside the image in an artefact repository.</p>
</li>
<li><p>Cosign signs the image using a key held in a Secrets Manager, never committed to the repository.</p>
</li>
<li><p>The image is pushed to the registry.</p>
</li>
<li><p>ArgoCD or Flux syncs the new manifest to the cluster in line with GitOps practices.</p>
</li>
<li><p>Admission control validates the incoming workload for image signatures, non-root user access, resource limits, and the absence of privileged escalation. I'll walk through exactly how in the next article.</p>
</li>
<li><p>If the workload passes admission, we'll schedule it. If not, it is rejected before a single container starts.</p>
</li>
<li><p>Falco continuously monitors the running workload for anomalous behaviour throughout the pod's lifetime.</p>
</li>
</ol>
<p>Each step here is automated and must be followed. Nobody should be able to bypass a signature check by asking nicely in a Slack channel.</p>
<h3>Best Practices and Common Mistakes:</h3>
<ul>
<li><p>Once you’ve tested a policy with real traffic, enforce it in “Enforce” mode, not just “Audit” mode. Teams that leave everything in audit mode are only collecting logs; they're not actively defending their systems.</p>
</li>
<li><p>Don’t treat SBOMs as just a compliance checkbox. When new CVEs are announced, make sure to actually query them. An unreviewed SBOM is just taking up space.</p>
</li>
<li><p>Don’t skip the runtime layer just because admission control seems enough. Admission control can’t catch vulnerabilities that are exploited after a signed image is already running.</p>
</li>
</ul>
<h3>Future Trends:</h3>
<p>Kubernetes security is moving towards more automation and stronger verification.</p>
<ul>
<li><p><a href="https://www.sigstore.dev/"><strong>Sigstore</strong></a> makes image signing easier by removing the need to manage long-term signing keys.</p>
</li>
<li><p><a href="https://slsa.dev/"><strong>SLSA</strong></a> helps teams prove software was built securely and in a trusted way.</p>
</li>
<li><p><a href="https://github.com/in-toto/in-toto"><strong>in-toto</strong></a> verifies every step of the software delivery pipeline, from source code to deployment.</p>
</li>
<li><p><a href="https://spiffe.io/"><strong>SPIFFE</strong></a> <strong>and SPIRE</strong> give each workload its own secure identity rather than relying on IP addresses.</p>
</li>
</ul>
<p>These technologies are the next step in Kubernetes security. The specific tools may change, but the goal stays the same: verify everything, trust nothing by default, and secure every part of the software supply chain.</p>
<h3><strong>References</strong></h3>
<ul>
<li><p>Kubernetes documentation, <a href="http://kubernetes.io/docs"><strong>kubernetes.io/docs</strong></a></p>
</li>
<li><p>CNCF, Cloud Native Security Whitepaper, <a href="http://cncf.io"><strong>cncf.io</strong></a></p>
</li>
<li><p>Sigstore project, sigstoredev</p>
</li>
<li><p>SLSA framework, slsadev</p>
</li>
<li><p>in-toto project, <a href="http://in-toto.io"><strong>in-toto.io</strong></a></p>
</li>
<li><p>SPIFFE/SPIRE, <a href="http://spiffe.io"><strong>spiffe.io</strong></a></p>
</li>
<li><p>Trivy documentation, <a href="http://aquasecurity.github.io/trivy"><strong>aquasecurity.github.io/trivy</strong></a></p>
</li>
<li><p>Kyverno documentation, <a href="http://kyverno.io"><strong>kyverno.io</strong></a></p>
</li>
<li><p>OPA Gatekeeper documentation, <a href="http://open-policy-agent.github.io/gatekeeper"><strong>open-policy-agent.github.io/gatekeeper</strong></a></p>
</li>
<li><p>Falco documentation, <a href="http://falco.org"><strong>falco.org</strong></a></p>
</li>
</ul>
<p><strong>Next in this series:</strong> Enforcing Kubernetes Security with Kyverno and OPA Gatekeeper</p>
]]></content:encoded></item></channel></rss>