Kubernetes Security Hardening – Image Whitelisting

Radeeka Kaththirige
9 min read | Posted on 13 Oct, 2023
share
News - Q4US

Background

We are using Docker containers in Kubernetes environments to host all our workloads. This required us to start looking into security hardening in kubernetes clusters. Securing a Kubernetes cluster requires implementing multiple layers of security controls, including network security, access controls, container security, and runtime security. Here are some general best practices to harden the security of a Kubernetes cluster.

1. Use a secure API server:

The Kubernetes API server is the central point for managing the cluster. Ensure that it is configured securely and accessible only through TLS. Use Role-Based Access Control (RBAC) to control access to the API server.

2. Implement network security:

Use network policies to restrict network traffic between pods and nodes in the cluster. Configure the Kubernetes API server to only listen on the internal network interface and restrict access to the Kubernetes API server and etcd server to only authorized personnel.

3. Use secure container images:

Ensure that only trusted container images are used in the cluster. Use image scanning tools to check for vulnerabilities in container images.

4. Implement access controls:

Use RBAC to limit access to the Kubernetes API server and other critical resources. Implement least privileged access controls to limit the access of users and services to only the resources they need.

5. Use secrets management:

Use Kubernetes secrets to store sensitive information such as passwords, certificates, and API keys. Avoid storing secrets in plain text, and encrypt sensitive data at rest and in transit.

6. Monitor the cluster:

Implement logging and monitoring to detect any suspicious activities in the cluster. Use tools like Prometheus and Grafana to monitor the health of the cluster and detect any anomalies.

7. Regularly update and patch the cluster:

Stay up-to-date with the latest Kubernetes patches and updates to ensure that the cluster is secure and free from vulnerabilities.

These are some general best practices to harden the security of a Kubernetes cluster. Implementing these controls can help you secure your Kubernetes cluster against potential security threats. Now let us look at how to make use of above 3rd point, to use only secure container images in a cluster.

Design and Implementation

In a Kubernetes cluster, you can restrict container images by using admission controllers. Kubernetes admission controller is a component of the Kubernetes API server that intercepts requests to the Kubernetes API before they are persisted in the etcd datastore. Admission controllers can modify, validate, or reject requests based on policies and rules. These can be either built-in or custom. Built-in admission controllers are included with Kubernetes and are enabled by default. Custom admission controllers can be developed to implement specific policies and rules.

First, let us look at the idea behind this and how it works by breaking it down as follows.

1. Enable the “ImagePolicyWebhook” admission controller: This admission controller enables you to validate container image references before a pod is created. You can enable this controller by adding it to the “kube-apiserver” arguments.

2. Create a webhook server: Create a webhook server that can validate the container images. The webhook server should receive an image reference from the “ImagePolicyWebhook” admission controller, validate it against a list of allowed images, and return a response indicating whether the image is allowed or not.

3. Configure the webhook server: Configure the webhook server to allow or deny container images based on a list of allowed images. You can also use a blacklist to deny specific images or a whitelist to allow specific images.

4. Create an image policy: Create an image policy that defines the rules for container images. The image policy should specify the webhook server to use allowed images.

Below is a graphical representation of the above flow.

One of the challenges with the above approach is the need to write our own webhook that serves our requirement. Kubernetes is quite an extensive tool so there are multiple language options that can be chosen to write one. It just needs to receive an incoming https request from the API server, perform a logic based on the image scanner and the security policy and respond back with a https response. In addition, we will need to setup the PKI for the communication between the API server and the webhook.

Open Policy Agent (OPA) – Gatekeeper is an open source project by CNCF, and a perfect candidate for this requirement. Below is a high-level overview of how it operates.

Here, whenever the API server receives a request, it will be passed to the OPA Gatekeeper via the admission controller in which the OPA GK will then does the webhooks job. The main advantage of this is, we don’t need to write any code, just setup the gatekeeper (which runs as a kubernetes deployment) and pass the policy template and policy constraints as kubernetes CRDs.

Now let’s look at how to implement this in a kubernetes cluster. I’ve already deployed a kubernetes cluster and I want my cluster to use only a list of whitelisted images inside the cluster.

Install OPA Gatekeeper

Use below command to deploy the OPA GK agent in our cluster which then will spin-up several resources related to the same.

kubectl apply -f https://raw.githubusercontent.com/open-policy-
agent/gatekeeper/master/deploy/gatekeeper.yaml

Next step will be to create the “Constraint Template” and “Constraint Policy” objects to enforce the CRD-Based policy to whitelist the images of our preference.

Policy Template

A Constraint Template is a type of resource used to define a set of constraints that can be used to validate Kubernetes resources. This contains a set of parameters and a Rego policy that specifies the constraints that should be enforced. The parameters can be used to configure the behavior of the Rego policy. Below is a sample constraint template manifest file.

constraint-template-crd.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8swhitelistedimages
spec:
  crd:
    spec:
      names:
        kind: k8sWhitelistedImages
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            images:
	type: array
	items: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8swhitelistedimages
        whitelisted_images = {images |
            images = input.parameters.images[_]
        }
 
        images_whitelisted(str, patterns) {
            image_matches(str, patterns[_])
        }
 
        image_matches(str, pattern) {
            contains(str, pattern)
        }
        violation[{"msg": msg}] {
          input.review.object
          image := input.review.object.spec.containers[_].image
          name := input.review.object.metadata.name
          not images_whitelisted(image, whitelisted_images)
          msg := sprintf("pod %q has an invalid image %q. Please, contact your DevOps.", [name, image])
        }

Policy Constraint

A Constraint Object is an instance of a Constraint Template. It is used to enforce a set of constraints on a specific Kubernetes resource. This is created by referencing a Constraint Template and providing values for any parameters defined in the template. The resulting Constraint Object is then associated with a specific Kubernetes resource, either at the cluster, namespace, or resource level.

When a Constraint Object is applied to a Kubernetes resource, it runs the Rego policy defined in the associated Constraint Template against the resource. If the resource violates any of the constraints specified in the policy, the Constraint Object rejects the resource and prevents it from being created or updated.

And below is a sample constraint policy manifest file that’s associated with above constraint template.

constraint-policy-crd.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: k8sWhitelistedImages
metadata:
  name: k8senforcewhitelistedimages
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    images:
      # Images allowed to be used by the infrastructure services inside the kubernetes cluster
      - grafana/grafana
      - k8s.gcr.io/metrics-server-amd64
      - kubernetesui/dashboard
      - kubernetesui/metrics-scraper
      - openpolicyagent/gatekeeper
      - prom/alertmanager
      - prom/node-exporter:v1.3.1
      - prom/prometheus
      - quay.io/coreos/kube-state-metrics
      - quay.io/kubernetes-ingress-controller/nginx-ingress-controller
      - registry.k8s.io/coredns/coredns:v1.9.3
      - registry.k8s.io/etcd:3.5.4-0
      - registry.k8s.io/kube-apiserver:v1.25.0
      - registry.k8s.io/kube-controller-manager:v1.25.0
      - registry.k8s.io/kube-proxy:v1.25.0
      - registry.k8s.io/kube-scheduler:v1.25.0
      - weaveworks/weave-kube:latest
      - weaveworks/weave-npc:latest

We can deploy these CRDs in the declarative approach.

kubectl apply -f constraint-template-crd.yaml 
kubectl apply -f constraint-policy-crd.yaml

These should be applied in the cluster shortly after. Use the following command to verify the status.

kubectl get constrainttemplate,constraint
_____________________________________________________________________________________________________________________
NAME                                                              AGE
constrainttemplate.templates.gatekeeper.sh/k8swhitelistedimages   27h
 
NAME                                                                         ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8swhitelistedimages.constraints.gatekeeper.sh/k8senforcewhitelistedimages                        0

If there are any violations, it’ll be detected under “Total-Violations“. To see the violations in detail, can use below command.

kubectl get constraint k8senforcewhitelistedimages -o yaml 

Lastly, if we try to run a pod using any image that is not whitelisted in the policy constraint object, this is what happens.

root@server:~# kubectl run test1 --image=nginx
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [k8senforcewhitelistedimages] pod "test1" has invalid image "nginx". Please, contact your DevOps.
root@server:~#

This way, we can make sure that the cluster will only contain workloads from the images that are allowed by above CRD.

Note:

If you ever come across to implement this in a system which is having a large number of images, you can first deploy the OPA GK and the Policy Constraints to get the list of images it already has.

kubectl get constraint -o yaml k8senforcewhitelistedimages | grep "has invalid image" | awk -F '"' '{print $4}' | sort -n | uniq

Apart from this, can use a jsonpath query as well.

And then include the list of images inside your policy constraint CRDs.

In conclusion, implementing image whitelisting through OPA Gatekeeper in your Kubernetes environment is a crucial step towards improving the security of your containerized applications. By enforcing strict policies that allow only approved and trusted container images to be deployed, you significantly reduce the attack surface and mitigate the risk of potential vulnerabilities or exploits. This approach aligns with the principle of ‘defense in depth,’ adding a layer of protection to your Kubernetes clusters. As security challenges change over time, taking steps to protect your system upfront, like approving specific images, becomes even more crucial. By adopting OPA Gatekeeper for image validation, you ensure that your Kubernetes workloads are secured against unauthorized and potentially malicious container images. As you work on making your Kubernetes security better, using OPA Gatekeeper to approve images should definitely be a key part of your security plan.

Chat close - Q4US

We’re here for you

Can we help you with anything?

Let’s talk