Enhance Kubernetes Security with OPA Gatekeeper

The 22nd day’s post of the Mercari Advent Calendar 2020 is brought to you by @moricho, a Software Engineer Intern from the Microservices Platform Infra team.

Introduction

At Mercari, we run our microservices on GKE (Google Kubernetes Engine). As the engineering organization expands and the number of microservices increases, it becomes more challenging to ensure container related security. We can’t put all the security responsibility on the developers who develop microservices, and it is also difficult to ensure security with only people in the security team. As a platform provider, it is our responsibility to ensure we have basic guardrails in our system.

We use OPA Gatekeeper to enforce Kubernetes security policies, and last quarter, I worked on it to introduce new policies. In this post, we will learn about Gatekeeper and some of the specific security issues we’ve resolved using it. We’ll also take a look at these from a DevSecOps perspective.

Overview of OPA Gatekeeper

At first, OPA (Open Policy Agent) is an open source, general-purpose policy engine that enables unified, context-aware policy enforcement. And Gatekeeper is a tool hosted by OPA, which helps enforce policies and strengthen governance in Kubernetes environments. We can create policies in its own language called Rego. After creating policies, we deploy them on a kubernetes cluster as CRDs.

For example, we can enforce policies like:

• Limit the container registries to only allowed certain ones
• Enforce pods to have resource limits
• Restrict the addition of Linux capabilities to containers

Gatekeeper consists of two components — Gatekeeper Audit and Gatekeeper Controller Manager. Gatekeeper Audit audits existing resources on a kubernetes cluster periodically if any resources violate the policies we created. And Gatekeeper Controller Manager validates kubernetes resources based on policies when they are created or updated.

With OPA Gatekeeper and policies, we can validate kubernetes resources and ensure container security automatically without human check and without delegating security responsibilities to developers. It also enables developers to focus more on application logic.

Security risks in containers and mitigations

In this section, I will introduce some of our Gatekeeper policies which we implemented recently to enforce security best practices in our Kubernetes workloads.

Additional Capabilities

Containers that have extra capabilities are exposed to security risk. Docker has already blocked some capabilities such as SYS_ADMIN, NET_ADMIN, and so on, but thoughtlessly adding other capabilities will have unexpected security risks like container escaping. A vulnerability related to Linux capabilities was recently discovered in the Linux kernel, described in CVE-2020-14386, that may allow escape from a container to obtain root privileges. This CVE is related to CAP_NET_RAW capability, which allows ICMP traffic between containers, and the ping command also uses it. This capability enables ARP (Address Resolution Protocol) spoofing attacks. That exploits the mechanism of correlating IP addresses with MAC addresses to let attackers fake their identity and makes a system to forward packets to them.

So we created a policy that prevents adding extra capabilities. This policy checks PodSpec definitions in all Pod, Deployment, DaemonSet, and so on. In specific, it checks .spec.containers[].securityContext.capabilities.add in containers and init containers configurations. Of course, we also create a whitelist to allow resources that have to add capabilities. This requires Platform team and Security team approval.

spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      capabilities:
        add: ["NET_ADMIN", "SYS_TIME"]  <- should be deleted

Host Network

Containers using the host network can do many things to jeopardize our cluster:

• Allow processes in the pod to talk on the host’s loopback adapter
• Create port conflicts with other containers
• Create holes in our security model

Actually, CVE-2020-15257 disclosed on November 30, 2020, reported that containerd containers running in the host network namespaces with UID 0 allowed to gain the host root privileges through containerd’s abstract sockets, which is a kind of UNIX socket. Any container running with hostNetwork: true and UID 0 could connect to these sockets and call arbitrary API methods because containerd was exposing these sockets without authentication except checking UID.
The vulnerability has already been fixed, but running containers in the host network namespace with UID 0 is still dangerous.

So it’s better to have a policy to restrict the usage of the host network. Our policy checks PodSpec definition and limits the value of .spec.hostNetwork to false.

spec:
  hostNetwork: true  <- should be deleted
  containers:
  - name: nginx
    image: nginx

Host Path

Containers using volumes that mount host paths can read and write any files on their host, including confidential files. And also, they can escape namespace constraints and gain access to Pods in any other namespaces.

So we created a policy to restrict the usage of the host path volumes. This policy checks if pods use volumes whose type is hostPath.

spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - mountPath: /test-path
      name: host-volume  <- should not use
  volumes:
  - name: host-volume  <- should not use
    hostPath:
      path: /data
      type: Directory

Privileged Containers

Privileged mode allows container processes to access host devices. Host device access enables those processes to behave like host processes (running outside a container). In doing so, the privileged mode allows bypassing almost all Kubernetes and Docker security controls.

So we create a policy to prevent containers from running in privileged mode. This policy checks PodSpec definition and restricts the value of .spec.containers[].securityContext.privileged to false.

spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      privileged: true  <- should be deleted

As per writing this blog, we have dozens of these Gatekeeper policies which are helping us in keeping security best practices in place. Not only security, Gatekeeper has also become our go to tool to create platform wide guardrails. Anytime we want to restrict something e.g, dedicated node-pool for a namespace, Gatekeeper is the first thing which comes in our mind.

Promote DevSecOps

The content introduced in the previous section is based on the very concept of DevSecOps that we often hear about these days. You can read about DevSecOps in the Mercari Advent Calendar 15th article if you want to learn more about it.

DevSecOps, as the name implies, adds Continuous Security to DevOps. Continuous Security spans the entire software development life cycle, enabling you to continuously and smoothly enhance security.

In our Gatekeeper example, we perform validation when each microservice team deploys their service to the Kubernetes cluster in the development environment. That means automatic and continuous security checks during the deployment phase of the development cycle. This way, you can find and fix problems early on in your daily development.
It also speeds up releases by reducing the chance of finding a significant bug or vulnerability just before release that we need to fix as soon as possible. That is exactly what the fundamental idea in DevSecOps is: Shift-Left Security.

We also asked the Security team to help us from the planning stage to determine what solution to take for what security risk, and we proceeded with the plan under their review.
The collaboration between our platform team, which is responsible for increasing developer productivity, and the security team is essential to promote DevSecOps.

Wrap Up

Through this blog post, I introduced OPA Gatekeeper and some security issues we resolved.
Suppose security issues occur and confidential information is leaked. In that case, we will need huge money to make up for it, and also our customers will lose confidence in our product and company. So I think it is essential in the long term to enhance our platform security to gain a sense of trust in the product and to prepare for attackers. Besides that, it’s worthwhile to enhance developer productivity by enabling developers to focus on application logic without getting distracted by security issues.
I hope it helps you think about container/kubernetes security.

Mercari is looking for a Microservices Platform Engineer who shares our mission and values.

Tomorrow’s blog post —the 23rd in the Mercari Advent Calendar 2020— will be written by Rishabh -san. I hope you are looking forward to it!