One of the most important developments in Information Technology is clearly the rapid increase of companies deploying Kubernetes and Docker Containers.
However, with this rapid increase comes a significant number of organizations not clearly managing their container environments. This poor oversight can lead to numerous container threats that increase the risks for your organization.
Why not go into the new year and be a hero for your customers and organizations by working on the improvement of their security posture.
Kubernetes is a great place to start for improving security and in this post Ill be covering the Top 12 Kubernetes Security Best Practices for 2020!
Lets get started!
Enable Role-Based Access Control (RBAC)
Starting in Kubernetes 1.6 RBAC policies are actually enabled by default.
RBAC policies are critical to manage your cluster since they allow you to specify which types of actions are permitted. (Who, What, When, etc)
For Example — Google Cloud IAM and Kubernetes RBAC have similarly named roles (cluster admin, admin, developer/edit, and viewer)
- Being Passive is the culprit of most of the common vulnerabilities.
- Think thru your enterprises permissioning and validate RBAC is still enabled.
- Don’t enable cluster wide permissions use namespace. (Everyone is a guest)
- Force Cluster Authentication
- Commands to know – kubectl get clusterrolebinding and – kubectl get rolebinding –all-namespaces
For those like really cool tools then give Kudos to Fairwinds for providing the RBAC Tool Lookup
RBAC Lookup is a CLI that allows you to easily find Kubernetes roles and cluster roles bound to any user, service account, or group name.
Use Namespaces to Establish Enterprise Security Boundaries
- Identifying and clearly segmenting your workloads is common sense.
- Validate your namespace
- Validate your network policies
- Commands to know- kubectl get namespaces and – kubectl get pods –all-namespaces
Validate and enhance your cluster node’s security by being guarded…
- Upgrade nodes routinely
- Lock down admin accounts, Restrict access to etcd
- Principle of least privilege
- Lock down ports (Resource Limits and Quotas) need to be identified during account audits)
- The link below clearly goes through this as a great reference
If your weak on containers consider getting enabled with Linux Foundation. https://shareasale.com/r.cfm?b=1481901&u=2037411&m=59485&urllink=&afftrack=
Audit Logging – Step it up if your already performing this and if your not then shame on your technical leadership since there really is no good not to audit.
- Auditing is a best practice for most aspects of IT services
- Auditing monitoring logins, failed logins, access violations, etc.
- Setup up monitoring to alert when these types of violations occur.
- Identify credential/role creep
- Advanced Persistent Threat thrive on auditing being neglected.
Understand and confirm Cloud Based Metadata services. Whether your using Google Cloud Kubernetes Engine or AWS Containers you have options on how to address metadata.
- Metadata is data about data.
- Used in Cloud services for identify services, monitoring, billing, auditing, etc.
- AWS, GCP and Azure all support Metadata in detail
- Metadata could be exploited especially when its clear.
- Enable cloud vendors concealment capacity around metadata.
Monitoring your Cloud-Native Applications
- Monitoring should be a best practice that every uses.
- Experience has shown me otherwise.
- Monitoring is needed not just for hardware cluster.
- Abstractions can be missed for example – Monitor pods, services and replicas in your container environment.
- For example Google Cloud Kubernetes Engine supports Kubernetes based monitoring and Stackdriver legacy.
Use Managed Services when Possible
- Managed serviced are cloud based services
- Provider handles cluster node maintenance. (varying degrees in a shared security model)
- Providers have a suite of tools that enhance you DevOps pipelines.
- Consider a Hybrid deployment (On premises and Cloud)
- Google Kubernetes Engine, Amazon Container Engine are excellent managed services.
Prevent Kubelet from changing label keys ( An agent that runs on each node in the cluster. It makes sure that containers are running in a pod.)
- Node authorization is a special-purpose authorization mode that specifically authorizes API requests made by kubelets
- Consider the fact that prevented nodes cannot use their Kubelet credentials to label their node object and actually schedule pods – NodeRestriction
- Node Authorizer page to review https://kubernetes.io/docs/reference/access-authn-authz/node/
Use the new feature called Ephemeral Storage
- Kubernetes 1.8 new resource which is called Ephemeral storage.
- Containers can use ephemeral storage for local storage.
- Configured local ephemeral storage? Check to see that you have set requests and limits for each container for this resource type.
- Command to know – kubectl describe pod -n <namespace_name> <pod_name>
- For example in Google Cloud the feature is in beta mode.
- Beginning in GKE version 1.10, you can manage your local ephemeral storage resources as you do your CPU and memory resources. System reservations for local storage are made primarily for disk space used by container images.
- More info https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#local_ephemeral_storage
Use Etcd Snapshots consistently which provide Performance, Stability and Security for your environment. (Consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.)
- Etcd snapshots are backups of the etcd cluster which of course would provide for cluster disaster recovery
- Dedicated Hardware to minimize resource starvation
- Perform correct number of snapshots – Set snapshots- etcd –snapshot-count=X (etcd v3.2 = Default 100,000)
- Reference page below for more information
Design a Highly Available cluster (Orchestration Layer)
- Multi Master Clusters Min. of 3 master nodes distributed across multiple zones.
- Worker nodes need to be distributed across zones for HA
- Distributing master nodes across zones which ensures that master nodes are immune to outages of entire availability zones
- Commands to know –master-zones flag and ensure Etcd – Replicate – 5 etcd members for production clusters.
Training your team! Never expect someone to know what they never learned. Investing in your team is critical.
Training your team to deploy containers and applications in a secure manner is one of the best ways to mitigate issues. Many organizations such as O’Reilly, Coursera, Linux Foundation and Quickstart have execellent courses and numerous options available. Not to mention there are numerous free resources from many vendors.
Considering taking the Certified Kubernetes Admin exam (CKA). Check out this resource.
Pre Order my book on Amazon! Architecting Enterprise Blockchain Solutions from Wiley Sybex.
Carry on my cloud friends and please do let me know any feedback or suggestions.
Joe Holbrook, the Cloud Tech Guy