8 ways to bolster Kubernetes security

As an orchestration platform, Kubernetes can affect many runtime security functions, including authentication, authorization, logging, and resource isolation. Since it also affects the container runtime environment, it's a crucial part of maintaining a secure container infrastructure.

Researchers have found hundreds of Kubernetes administration consoles accessible over the Internet without even password protection. Hackers have used these consoles to access Amazon Web Services and Microsoft Azure environments and use those resources to mine cryptocurrencies.

Here are eight tips for addressing this and other issues to keep Kubernetes secure.

Application Security Research Update: The State of App Sec in 2018

1. Protect access to your Kubernetes dashboard

Aqua Security is seeing a lot of searching for unsecured, open-access Kubernetes dashboards, said Liz Rice, technology evangelist at the maker of a container security platform.

And there's a motive, she added. "There are search tools out there that people can use to find open ports across the Internet and exploit them for exfiltrating data or to obtain resources for things like cryptocurrency mining."

Maya Kaczorowski, product manager of Google Cloud, recommends turning off the Kubernetes web user interface entirely. "The Kubernetes web UI uses a highly privileged service account," she explained. "In the Google Kubernetes Engine, this is disabled by default since version 1.7."

2. Lock down administrative access

Make sure anyone who signs on as an administrator has credentials. The same is true for anyone with access to the Kubernetes API.

"[Having] access to the API is like having root access on a server, so making sure that you need to be authenticated is really, really critical."
Liz Rice

Kubernetes offers both role-based access control (RBAC) and attribute-based access control (ABAC). Google's Kaczorowski recommended using RBAC and disabling ABAC. This allows you to give permissions based on predefined roles and privileges, she explained.

3. Keep close tabs on where credentials are located

This isn't just a Kubernetes problem, but a nettlesome challenge for all organizations. It's really easy for people to leave their credentials in places where they shouldn't be left, Rice said.

"People are leaving credentials in GitHub repositiories or leaving them in Docker container images. If a location is publicly accessible, then those credentials are going to be accessible to the world."
—Liz Rice

4. Use trusted content

If a single container is defective, Kubernetes will make the problem worse by deploying it across many machines. Many times those vulnerabilities enter a container by pulling down images from a public hub, such as Docker Hub, which aren't validated and signed, said George Gerchow, vice president of security and compliance at Sumo Logic, a cloud-based log management and analytics company.

"A lot of organizations just getting into containers don't realize that's taken place, and they don't know how to validate or sign what's coming from Docker Hub."
George Gerchow

Just as most organizations use open-source components when building applications, many use prebuilt images for their containers. "When you look at public container images, you want to start with trusted content," said Kirsten Newcomer, a security strategist at Red Hat, an open-source software provider. For example, you want to make sure any images you use are signed so you can verify their provenance. If possible, obtain images from a source that can give you vulnerability data on the image and that updates its images when new vulnerabilities are discovered.

To protect your container images, Kaczorowski recommends using private registries, maintaining standard base images for developers to use, and scanning images for security vulnerabilities.

John Morello, CTO at software container security company Twistlock, said it's important to get administrators in the game. "It's important for the administrator to take a proactive approach when managing cluster vulnerabilities."

"This means utilizing vulnerability assessment and management tools throughout the lifecycle of the application, the environment, and the deployment process, to ensure that only robust code is deployed and that the environment is free of known defects and configuration vulnerabilities."
John Morello

5. Recreate container images after patching

Patch containers by rebuilding them, not by patching their components in place. That's a departure from what's typically done with an application or virtual machine. With a VM, for instance, you may have to give a person administrative privileges to access the VM and apply patches to the system library and application in the runtime environment. "Best practices for containers is not to do that," said Red Hat's Newcomer.

Rebuilding an image has some security advantages, Newcomer explained. 

"You never have to give anyone root access to your runtime environment."
Kirsten Newcomer 

It's also a more efficient way to deploy container changes, because Kubernetes can automate the deployment much faster than you can patch vulnerabilities one container at a time.

6. Never let a container run as root

If a container with root access is compromised, an adversary will have unbridled access to a deployment. "When you don't run things as root, you take away some of the big threat vectors," said Chenxi Wang, founder of the Jane Bond Project, a cybersecurity consultancy.

None of your application containers should be running as root, Newcomer added. "One of the nice things about containers is applications work within their own namespace in the container, so it should almost never need root access outside the container."

Sumo Logic's Gerchow said the risk is much greater with Kubernetes versus Docker.

"That's the scary part about Kubernetes. With Docker, you're running root in containers one by one. With Kubernetes, you're deploying in the thousands."
—George Gerchow

Kubernetes can prevent a container from gaining root access through its security context constraints feature. "If whoever built the container image wasn't careful and has it set up to run as root, a security context constraint can be used at deploy time to stop deployment of that container," Red Hat's Newcomer explained.

The orchestrator is the infrastructure that touches everything, so it's a lot easier to enforce a policy with the orchestrator versus doing it manually, Wang added.

"Kubernetes is a way to increase automation. It's also a place where your get increased visibility and compliance status, so it's a very effective place to enforce your security policy."
Chenxi Wang

7. Use Linux's container-specific security features

To protect your containers once they're up and running, Google's Kaczorowski recommended using Linux-specific container security features, such as namespaces to contain workloads, cgroups to contain resource use, and seccomp to limit what a user can call.

"You want to make sure that the underlying operating system knows how to work with containers," Newcomer explained. "You want an operating system that can secure containers at the boundaries, securing the host kernel from container escapes and securing containers from each other."

Linux can do that, she added. "It assures that you're creating appropriate container isolation."

8. Encrypt Kubernetes etcd

Kubernetes stashes critical information for its operation in a key-value store called etcd. Prior to version 1.7 of the orchestrator, it stored all its data in plaintext. That meant that anyone who gained access to the etcd could read all the important information stored there, including Kubernetes secrets and API tokens. That changed with version 1.7. "You can now encrypt that etcd environment," Newcomer said.

New kid on the block

Since its introduction, Kubernetes has continued to enhance its security features. "Each successive release has more and more security controls built in," Wang said. 

"They get richer and richer, but Kubernetes is so new that lots of practitioners don't know what security controls come with it. So one of the first things to do is study up on what controls are there and use them to strengthen your security posture."
—Chenxi Wang

Share your team's best practices for securing Kubernetes in the comments below.

Topics: Security