Background
We are using Docker containers in Kubernetes environments to host all our workloads. This required us to start looking into security hardening in Kubernetes clusters. Securing a Kubernetes cluster requires implementing multiple layers of security controls, including network security, access controls, container security, and runtime security. Here are some general best practices to harden the security of a Kubernetes cluster.
- Use a secure API server:
The Kubernetes API server is the central point for managing the cluster. Ensure that it is configured securely and accessible only through TLS. Use Role-Based Access Control (RBAC) to control access to the API server.
- Implement network security:
Use network policies to restrict network traffic between pods and nodes in the cluster. Configure the Kubernetes API server to only listen on the internal network interface and restrict access to the Kubernetes API server and etcd server to only authorized personnel.
- Use secure container images:
Ensure that only trusted container images are used in the cluster. Use image scanning tools to check for vulnerabilities in container images.
- Implement access controls:
Use RBAC to limit access to the Kubernetes API server and other critical resources. Implement least privilege access controls to limit the access of users and services to only the resources they need.
5. Use secrets management:
Use Kubernetes secrets to store sensitive information such as passwords, certificates, and API keys. Avoid storing secrets in plain text, and encrypt sensitive data at rest and in transit.
6. Monitor the cluster:
Implement logging and monitoring to detect any suspicious activities in the cluster. Use tools like Prometheus and Grafana to monitor the health of the cluster and detect any anomalies.
7. Regularly update and patch the cluster:
Stay up-to-date with the latest Kubernetes patches and updates to ensure that the cluster is secure and free from vulnerabilities.
These are some general best practices to harden the security of a Kubernetes cluster. Implementing these controls can help you secure your Kubernetes cluster against potential security threats. Now let us look at how to make use of the above 3rd point, to use only secure container images in a cluster.
Design and Implementation
In a Kubernetes cluster, you can restrict container images by using admission controllers. Kubernetes admission controller is a component of the Kubernetes API server that intercepts requests to the Kubernetes API before they are persisted in the etcd datastore. Admission controllers can modify, validate, or reject requests based on policies and rules. These can be either built-in or custom. Built-in admission controllers are included with Kubernetes and are enabled by default. Custom admission controllers can be developed to implement specific policies and rules.
First, let us at the idea behind this and how it works by breaking it down as follows.
- Enable the “ImagePolicyWebhook” admission controller :
This admission controller enables you to validate container image references before a pod is created. You
can enable this controller by adding it to the “kube-apiserver” arguments.
- Create a webhook server :
Create a webhook server that can validate the container images. The webhook server should receive an image reference from the “ImagePolicyWebhook” admission controller, validate it against a list of allowed images, and return a response indicating whether the image is allowed or not.
- Configure the webhook server :
Configure the webhook server to allow or deny container images based on a list of allowed images. You can also use a blacklist to deny specific images or a whitelist to allow specific images.
- Create an image policy :
Create an image policy that defines the rules for container images. The image policy should specify the webhook server to use allowed images.
Below is a graphical representation of the above flow.
One of the challenges with the above approach is the need to write our own webhook that serves our requirement. Kubernetes is quite an extensive tool so there are multiple language options that can be chosen to write one. It just needs to receive an incoming https request from the API server, perform a logic based on the image scanner and the security policy and respond back with a https response. In addition, we will need to setup the PKI for the communication between the API server and the webhook.
Open Policy Agent (OPA) – Gatekeeper is an open source project by CNCF, and a perfect candidate for this requirement. Below is a high-level overview of how it operates.
Here, whenever the API server receives a request, it will be passed to the OPA Gatekeeper via the admission controller in which the OPA GK will then does the webhooks job. The main advantage of this is, we don’t need to write any code, just setup the gatekeeper (which runs as a Kubernetes deployment) and pass the policy template and policy constraints as Kubernetes CRDs.
Now let’s look at how to implement this in a Kubernetes cluster. I’ve already deployed a Kubernetes cluster and I want my cluster to use only a list of whitelisted images inside the cluster.
Install OPA Gatekeeper
Use below command to deploy the OPA GK agent in our cluster which then will spin-up several resources related to the same.
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yamlNext step will then do the webhooks job.
Policy Template
A Constraint Template is a type of resource used to define a set of constraints that can be used to validate Kubernetes resources. This contains a set of parameters and a Rego policy that specifies the constraints that should be enforced. The parameters can be used to configure the behavior of the Rego policy.
Below is a sample constraint template manifest file.
constraint-template-crd.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: k8swhitelistedimages
spec:
crd:
spec:
names:
kind: k8sWhitelistedImages
validation:
# Schema for the `parameters` field
openAPIV3Schema:
properties:
images:
type: array
items: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8swhitelistedimages
whitelisted_images = {images |
images = input.parameters.images[_]
}
images_whitelisted(str, patterns) {
image_matches(str, patterns[_])
}
image_matches(str, pattern) {
contains(str, pattern)
}
violation[{"msg": msg}] {
input.review.object
image := input.review.object.spec.containers[_].image
name := input.review.object.metadata.name
not images_whitelisted(image, whitelisted_images)
msg := sprintf("pod %q has an invalid image %q. Please, contact your DevOps.", [name, image])
}
Policy Constraint
A Constraint Object is an instance of a Constraint Template. It is used to enforce a set of constraints on a specific Kubernetes resource. This is created by referencing a Constraint Template and providing values for any parameters defined in the template. The resulting Constraint Object is then associated with a specific Kubernetes resource, either at the cluster, namespace, or resource level.
When a Constraint Object is applied to a Kubernetes resource, it runs the Rego policy defined in the associated Constraint Template against the resource. If the resource violates any of the constraints specified in the policy, the Constraint Object rejects the resource and prevents it from being created or updated.
And below is a sample constraint policy manifest file that’s associated with the above constraint template.
constraint-policy-crd.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1 kind: k8sWhitelistedImages metadata: name: k8senforcewhitelistedimages spec: match: kinds: - apiGroups: [""] kinds: ["Pod"] parameters: images: # Images allowed to be used by the infrastructure services inside the kubernetes cluster - grafana/grafana - k8s.gcr.io/metrics-server-amd64 - kubernetesui/dashboard - kubernetesui/metrics-scraper - openpolicyagent/gatekeeper - prom/alertmanager - prom/node-exporter:v1.3.1 - prom/prometheus - quay.io/coreos/kube-state-metrics - quay.io/kubernetes-ingress-controller/nginx-ingress-controller - registry.k8s.io/coredns/coredns:v1.9.3 - registry.k8s.io/etcd:3.5.4-0 - registry.k8s.io/kube-apiserver:v1.25.0 - registry.k8s.io/kube-controller-manager:v1.25.0 - registry.k8s.io/kube-proxy:v1.25.0 - registry.k8s.io/kube-scheduler:v1.25.0 - weaveworks/weave-kube:latest - weaveworks/weave-npc:latest
We can deploy these CRDs in the declarative approach.
kubectl apply -f constraint-template-crd.yaml kubectl apply -f constraint-policy-crd.yaml
These should be applied in the cluster shortly after. Use the following command to verify the status.
kubectl get constrainttemplate,constraint _____________________________________________________________________________________________________________________ NAME AGE constrainttemplate.templates.gatekeeper.sh/k8swhitelistedimages 27h NAME ENFORCEMENT-ACTION TOTAL-VIOLATIONS k8swhitelistedimages.constraints.gatekeeper.sh/k8senforcewhitelistedimages 0
If there are any violations, it’ll be detected under “Total-Violations“. To see the violations in detail, can use below command.
kubectl get constraint k8senforcewhitelistedimages -o yaml
Lastly, if we try to run a pod using any image that is not whitelisted in the policy constraint object, this is what happens.
root@server:~# kubectl run test1 --image=nginx Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [k8senforcewhitelistedimages] pod "test1" has invalid image "nginx". Please, contact your DevOps. root@server:~#
This way, we can make sure that the cluster will only contain workloads from the images that are allowed by above CRD.
Note:
If you ever come across to implement this in a system which is having a large number of images, you can first deploy the OPA GK and the Policy Constraints to get the list of images it already has.
kubectl get constraint -o yaml k8senforcewhitelistedimages | grep "has invalid image" | awk -F '"' '{print $4}' | sort -n | uniq
Apart from this, can use a jsonpath query as well.
And then include the list of images inside your policy constraint CRDs.
Certainly! Here’s a concluding paragraph you could consider for your blog on “Kubernetes Security Hardening – Image Whitelisting” using OPA Gatekeeper:
In conclusion, implementing image whitelisting through OPA Gatekeeper in your Kubernetes environment is a crucial step towards improving the security of your containerized applications. By enforcing strict policies that allow only approved and trusted container images to be deployed, you significantly reduce the attack surface and mitigate the risk of potential vulnerabilities or exploits. This approach aligns with the principle of ‘defense in depth,’ adding a layer of protection to your Kubernetes clusters. As security challenges change over time, taking steps to protect your system upfront, like approving specific images, becomes even more crucial. By adopting OPA Gatekeeper for image validation, you ensure that your Kubernetes workloads are secured against unauthorized and potentially malicious container images. As you work on making your Kubernetes security better, using OPA Gatekeeper to approve images should definitely be a key part of your security plan.
Leave a Reply