Bulletproof Your Kubernetes Clusters: Essential Security Best Practices
Kubernetes has become a vital component of software infrastructure in recent times. Most media to large companies are already using Kubernetes clusters for their workloads. As a DevOps engineer, you are most likely maintaining either an on-prem Kubernetes cluster or a PaaS such as Amazon EKS, Microsoft AKS, or GKE. However, irrespective of how you run your Kubernetes clusters, you must ensure they are secure.
Kubernetes API server has several layers of security to safeguard your infrastructure. The API communication is encrypted using TLS and valid certificates. Moreover, all API requests are authenticated using several mechanisms, and the authorized requests are validated by admission control modules, except read/get requests. Kubernetes offers multiple security options out of the box, but to make your infrastructure more secure, you must follow additional security best practices. Today we will discuss some of the most critical security best practices for Kubernetes.
1. Use Kubernetes Role-Based Access Control (RBAC)
Kubernetes supports multiple authorization models for its API server. These are ABAC (Attributed-Based Access Control), RBAC (Role-Based Access Control), Node authorization and the Webhook mode. Out of all these, RBAC is the more secure and most widely used and is ideal for enterprises and medium to large organizations. With RBAC, you can define role-based access control that closely resembles your organization's business rules. RBAC also works great with OIDC authentication.
Most Kubernetes distributions have RBAC enabled by default. You can check this by running the command kubectl cluster-info dump | grep authorization-mode, which should have authorization-mode=RBAC in the output. If not, you can enable it using the --authorization-mode flag for the API server when creating or patching the cluster. For example, setting --authorization-mode=RBAC, Node will enable both RBAC and Node authorization on the cluster.
Once RBAC is enabled, you can create roles (Role/ClusterRole) and bindings (RoleBinding/ClusterRoleBinding) to control access to your resources. Here is an example of a role and role binding that lets users view pods and services.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: fancy-namespace
name: pod-service-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods", "services]
verbs: ["get", "watch", "list"]
The below binds the above role to a specific group of users.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods-services
namespace: fancy-namespace
roleRef:
kind: Role #this must be Role or ClusterRole
name: pod-service-reader # this must match the name of the Role or ClusterRole you wish to bind to
apiGroup: rbac.authorization.k8s.io
subjects: # subject can be individual users or a group of users. Group is defined in the external authentication service, in this case, an OIDC server
- kind: Group
name: k8s-restricted-users
2. Secure the API server with OpenID Connect
Kubernetes supports many authentication mechanisms. Some of the most common are:
Client certificates
Basic authentication
Tokens (Service account tokens, Bearer tokens, and so on)
Proxy
Out of all these authentication mechanisms, OIDC is the most secure and scalable solution. It is ideal for clusters accessed by large teams as it provides a single sign-on solution for all users and makes it easy to onboard and offboard users. It is also way more secure than other mechanisms as you don't have to store any sensitive information on a user's computer, like client secrets and passwords. You can also use features like MFA and Yubikey if supported by your OIDC provider.
OIDC combined with RBAC becomes necessary as more and more people start accessing the cluster. It becomes essential to create groups and roles and provide limited access to specific groups. You can read more about this in my previous post How to Secure Your Kubernetes Cluster with OpenID Connect and RBAC.
3. Use Secrets for all sensitive data with appropriate access
This one should be a no-brainer. Kubernetes has a Secret resource that can be used to store sensitive data. This is a great way to store passwords, keys, and other sensitive data. Secrets can be used for storing string data, docker config, certificates, tokens, files, and so on. Secrets can be mounted as data volumes or exposed as environment variables to be used in containers. Secrets can be plain text or encoded, but please don't be that person who uses plain text secrets.
Secrets are flexible and native to Kubernetes, so there is no reason for you not to use them. Also, make sure to implement proper RBAC for secrets so that not everyone has access to them.
4. Keep Kubernetes version up to date
Like any other software, Kubernetes also has bugs and issues. And from time to time, there might be a high-severity bug that calls for a CVE. Hence, it's an excellent idea to keep the Kubernetes version up to date on the server and the CLI client. You can check the Kubernetes security and disclosure information website to see if there are known security vulnerabilities for your Kubernetes version. If you are using a managed PaaS, it should be pretty easy to upgrade, and for on-prem installations, there are tools like kOps, kubeadm, and so on, that make it easy to upgrade clusters.
5. Restrict kubelet, API, and SSH access
kubelet is the primary "node agent" running on each node, and by default, a cubelet's HTTP endpoints are not secured. This could allow unintended access and hence should be restricted.
When someone has access to a Kubernetes cluster, they can access the k8s API server and SSH into the cluster nodes themselves. To limit node access, cluster access should be limited as much as possible. Disable SSH access for non-admin users. Secure the API server using OIDC and RBAC, as we saw earlier so that only authenticated users with sufficient roles can access the API.
6. Secure container images
Securing the container images that run on the cluster is as important as securing the cluster itself. A malicious image running on a cluster could wreak havoc. Follow these best practices for container image security.
Do not run containers as root as this would give the container unlimited access to the host. Always run the containers using a non-root user.
Enable container image scanning in your CI/CD phase to catch known vulnerabilities using tools like clair or Anchore.
Use minimal up-to-date official base images and remove all unwanted dependencies, packages, and debugging tools from the image as it will make it more secure and lightweight.
Prevent loading unwanted kernel modules in the containers. These can be restricted using rules in
/etc/modprobe.d/Kubernetes-blacklist.confof the node or by uninstalling the unwanted modules from the node.Use official verified images for popular software. Use a trusted registry for non-official images and always verify the image publisher
Use Docker Bench for Security to audit your container images
Use Pod security policies to limit a container's access to the host further
You can read more about it in our "Container Security: A Developer Guide".
7. Control traffic between pods and clusters
Generally, pods within the same cluster will be able to communicate with each other, and if you have multiple clusters in the same network, there may be traffic between them as well. Do not leave this all open, as it could lead to a compromised cluster when another in the network is affected. Use Kubernetes network policies to control traffic between pods and clusters and allow only necessary traffic.
8. Use namespaces to isolate workloads
Do not run all your workloads in a single namespace. Isolating workloads in different namespaces based on business needs is more secure and easier to manage with RBAC. This way, you can fine-tune RBAC even further to let users access only what they need to see. You can also use Kubernetes network policies to isolate traffic between namespaces where applicable.
9. Limit resource usage
As with securing APIs and the cluster itself, it is also essential to set resource limits on how much CPU, memory, and persistent disk space is used by namespaces and resources. This secures your cluster from denial of service attacks when a particular container uses up all the resources. Resources quotas and limit ranges can be used to set limits at the namespace level, and Requests and limits can be used to set resource limits at the container level as well.
10. Use monitoring tools to monitor all traffic and enable audit logging
Finally, it is also extremely important to monitor and audit your clusters. Enable audit logging for the cluster and use monitoring tools to keep an eye on the networking traffic to, from and within a cluster. Monitoring can be done using open-source tools, like Prometheus, Grafana, or proprietary tools.
Bonus
Furthermore, keep these infrastructure best practices also in mind when securing your Kubernetes cluster.
Ensure that all communication is done via TLS.
Protect etcd with TLS, Firewall, and Encryption and restrict access to it using strong credentials.
Set up IAM access policies in a supported environment like a PaaS.
Rotate infrastructure credentials frequently.
Restrict cloud metadata API access when running in a PaaS like AWS, Azure, or GCP.
Learn more about Kubernetes and security
If you want to learn more about Kubernetes, OIDC, or using OIDC with Kubernetes, and security in general, check out these additional resources.
If you liked this tutorial, chances are you'll enjoy the others we publish. Please follow @wisdomcode_ on Twitter