Kubernetes users often ask me what they need to do to determine whether their deployment is running and configured safely from a security perspective. The best answer is to hire someone to perform penetration testing, but that can get expensive.
A quicker, cheaper option that can help you identify common configuration issues is to use the open-source, do-it-yourself penetration testing tool kube-hunter. (Disclosure: My company developed this tool.)
I don't want to discourage you from hiring a security company to run penetration testing on your deployments. Penetration testers use the same tools as hackers do to search for weaknesses in your deployment. They run many different types of tests to try to find exploitable weaknesses on all possible entry points.
After all, a live attacker searching for and exploiting a Kubernetes vulnerability is not a hypothetical scenario, so the best pen testers try to go beyond the most common attack scenarios; they learn from each test and adjust their approach.
Here's how to improve the hunt for security vulnerabilities in your Kubernetes deployments.
Kubernetes' flexibility brings complexity
In one early example of how systems could be exploited, electric vehicle manufacturer Tesla found itself in the news in February 2018 after discovering that its Kubernetes dashboard had been left open to the Internet with cluster administrator privileges. Someone discovered it, then used the cluster to run crypto-currency mining software at the company's expense.
Since that attack, major efforts have been dedicated toward making Kubernetes more secure out of the box. For example, now the dashboard isn't installed by default, and there are several effective security defaults built into the kubeadm installer. Nevertheless, with the flexibility of Kubernetes comes complexity. Even experienced users are at risk of deploying a system configured in a way that leaves them vulnerable.
Kube-hunter is not a replacement for comprehensive pen testing. But if you want to focus on evaluating your environment, kube-hunter allows you to "hunt" for security weaknesses in your Kubernetes clusters.
Let's talk network connections
Regular HTTP requests are open to monitoring by anyone. If you're using HTTPS, the network connection starts as regular HTTP and then gets promoted to HTTPS, which prevents you from running both HTTP and HTTPS on the same port. While HTTPS is more secure, you need a certificate to run the server end of an HTTPS connection.
Lots of server code, including some Kubernetes components, offers the option of using an insecure port so that a developer doesn't have to bother with the certificate configuration. In practice you don't ever want to run an insecure port in production—and you want to know that your cluster hasn't inadvertently left any insecure ports open. So this is something that kube-hunter looks for and reports on.
If you’re using HTTPS, the client may choose to identify itself over the request, and then the server can use that identity to figure out whether the client is authorized to make the request. For today's Kubernetes installations, that authorization is typically configured using role-based access control (RBAC).
When you make a request to the Kubernetes API without supplying any identity, it gets run under the system:anonymous service account. This can have permissions associated with it through RBAC, just as with any other service account. This means that by default some capabilities are available even to anonymous users.
Yes, it is possible to disable anonymous requests altogether, but that requires all clients to identify themselves, even if they only want to conduct health checks or service discovery. Because of this, you may consider it an acceptable risk to allow anonymous requests, but you should place highly restrictive RBAC policies on them. (This is how the current defaults are configured on an out-of-the-box installation from the kubeadm tool.)
Going threat hunting
Kube-hunter automatically searches for and flags open Kubernetes ports, relieving you of the burden of doing so manually. Start by running kube-hunter on any machine outside your cluster and, when prompted, give it the domain name or IP address of the cluster. You can also run kube-hunter on a machine in the cluster, and select the option to probe all the local network interfaces.
Another option is to run kube-hunter inside a pod in your cluster to simulate what attackers could do if they had compromised an application running in that pod. Every pod runs under a service account, and by default the code running in the pod—in this case, kube-hunter—has access to a token identifying that service account.
Kube-hunter can use the token to explore the API access that this service account token permits. The results depend on the RBAC settings for that pod's service account. So ideally you would run this in a pod under every possible service account, to get a view of what an attacker might be able to do if they were able to break into any one of the pods in your deployment.
By default kube-hunter will only run "passive hunters"—a series of tests that probe for potential access points (like open ports) within your cluster and perform read-only tests that won't change any state.
"Active hunting" can demonstrate that an exploit is possible, but also could potentially result in state-changing operations on the cluster. So use this option with care, perhaps only on non-production servers. For example, a passive hunter might try to obtain a list of pods; an active hunter might create a pod with a randomized name, and then (if successful) delete it again.
More hunters
Kube-hunter has a wide range of hunters that have the potential to uncover many common issues, and the list is increasing thanks to community contributions. You can find a list of the tests by running kube-hunter with the --list parameter.
The underlying code (which you can also run without using the website) is open source, and we encourage you to submit your own ideas and suggestions, alert the community to potential attacks, and even write your own hunters. Follow this link to the kube-hunter GitHub repo to get started.
One last caveat: Be responsible. Never use kube-hunter on other people's clusters.
Want to learn more? Come to my session, "Lessons from hacking Kubernetes with kube-hunter," at the O'Reilly Velocity Conference in San Jose, California. The conference runs June 10-13.
Keep learning
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed fast on the state of app sec testing with TechBeacon's Guide. Plus: Get Gartner's 2021 Magic Quadrant for AST.
Get a handle on the app sec tools landscape with TechBeacon's Guide to Application Security Tools 2021.
Download the free The Forrester Wave for Static Application Security Testing. Plus: Learn how a SAST-DAST combo can boost your security in this Webinar.
Understand the five reasons why API security needs access management.
Learn how to build an app sec strategy for the next decade, and spend a day in the life of an application security developer.
Build a modern app sec foundation with TechBeacon's Guide.
