Clever Geek Handbook

The ABCs of Security in Kubernetes: Authentication, Authorization, Auditing





Sooner or later, the operation of any system raises the question of security: ensuring authentication, separation of rights, auditing and other tasks. Many solutions have already been created for Kubernetes that can achieve compliance with standards even in very demanding environments ... The same material is devoted to the basic security aspects implemented within the framework of K8s built-in mechanisms. First of all, it will be useful for those who begin to get acquainted with Kubernetes - as a starting point for studying security issues.



Authentication



Kubernetes has two types of users:





The main difference between these types is that for Service Accounts there are special objects in the Kubernetes API (they are called ServiceAccounts



) that are tied to the namespace and set of authorization data stored in the cluster in objects of type Secrets. Such users (Service Accounts) are intended primarily for managing access rights to the Kubernetes API processes running in a Kubernetes cluster.



Ordinary Users do not have entries in the Kubernetes API: they must be managed by external mechanisms. They are intended for people or processes living outside the cluster.



Each request to the API is bound either to the Service Account, or to the User, or is considered anonymous.



User authentication data includes:





Kubernetes can use a large number of authentication mechanisms: X509 certificates, Bearer tokens, authentication proxies, HTTP Basic Auth. Using these mechanisms, a large number of authorization schemes can be implemented: from a static file with passwords to OpenID OAuth2.



Moreover, multiple authorization schemes are allowed at the same time. By default, the cluster uses:





The question about the management of ServiceAccounts is beyond the scope of this article, but I recommend starting to learn more about this issue from the official documentation page . We will consider in more detail the issue of the operation of X509 certificates.



Certificates for users (X.509)



The classic way of working with certificates involves:





After the above manipulations, a config of the form will be created in the .kube/config



file: 



 apiVersion: v1 clusters: - cluster: certificate-authority: /etc/kubernetes/pki/ca.crt server: https://192.168.100.200:6443 name: kubernetes contexts: - context: cluster: kubernetes namespace: target-namespace user: mynewuser name: mynewuser-context current-context: mynewuser-context kind: Config preferences: {} users: - name: mynewuser user: client-certificate: /home/mynewuser/.certs/mynewuser.crt client-key: /home/mynewuser/.certs/mynewuser.key


To facilitate the transfer of the config between accounts and servers, it is useful to edit the values ​​of the following keys:





To do this, you can encode the files indicated in them using base64 and register them in the config by adding the suffix -data



to the name of the keys, i.e. getting certificate-authority-data



, etc.



Certificates with kubeadm



With Kubernetes 1.15, working with certificates has become much easier thanks to the alpha version of its support in the kubeadm utility . For example, here is how the generation of a configuration file with user keys may now look: 



 kubeadm alpha kubeconfig user --client-name=mynewuser --apiserver-advertise-address 192.168.100.200


NB : The required advertise address can be viewed in the api-server config, which is located at /etc/kubernetes/manifests/kube-apiserver.yaml



by default.



The resulting config will be output to stdout. It must be saved in ~/.kube/config



user account or in the file specified in the KUBECONFIG



environment KUBECONFIG



.



Dig deeper



For those wishing to better understand the issues described:





Login



An authenticated account does not have permission to act in a cluster by default. Kubernetes has an authorization mechanism for granting permissions.



Prior to version 1.6, Kubernetes used an authentication type called ABAC (Attribute-based access control). Details about it can be found in the official documentation . This approach is currently considered legacy, but you can still use it at the same time as other types of authorization.



The actual (and more flexible) way of dividing cluster access rights is called RBAC ( Role-based access control ). It has been declared stable since Kubernetes 1.8 . RBAC implements a rights model that prohibits anything that is not explicitly permitted.

To enable RBAC , you need to run Kubernetes api-server with the --authorization-mode=RBAC



option. Parameters are set in the manifest with the api-server configuration, which is by default located on the path /etc/kubernetes/manifests/kube-apiserver.yaml



, in the command



section. However, RBAC is already enabled by default, so you probably should not worry about this: you can verify this by the value of authorization-mode



(in the already mentioned kube-apiserver.yaml



). By the way, among its values ​​there may be other types of authorization ( node



, webhook



, always allow



), but we will leave them outside the scope of the material.



By the way, we have already published an article with a fairly detailed story about the principles and features of working with RBAC, so I will further limit myself to a brief listing of the basics and examples.



The following API entities are used to control access to Kubernetes via RBAC:





Entities Role and RoleBinding are limited by namespace, i.e. must be within the same namespace. However, RoleBinding can refer to ClusterRole, which allows you to create a set of standard permissions and control access using them.



Roles describe rights using rule sets containing:





A more detailed discussion of authorization in Kubernetes can be found on the official documentation page. Instead, (or rather, in addition to this), I will give examples that illustrate its work.



RBAC Entity Examples



A simple Role



that allows you to get the list and status of pods and monitor them in the target-namespace



: 



 apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: target-namespace name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list"]


An example of ClusterRole



, which allows you to get a list and status of pods and monitor them throughout the cluster: 



 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: #  "namespace" ,   ClusterRole    name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list"]


An example of RoleBinding



, which allows the user mynewuser



"read" pods in the my-namespace



: 



 apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: read-pods namespace: target-namespace subjects: - kind: User name: mynewuser #     ! apiGroup: rbac.authorization.k8s.io roleRef: kind: Role #    “Role”  “ClusterRole” name: pod-reader #  Role,      namespace, #   ClusterRole,   #    apiGroup: rbac.authorization.k8s.io


Event Audit



Schematically, the Kubernetes architecture can be represented as follows:



image



The key component of Kubernetes, which is responsible for processing requests, is api-server . All operations on the cluster pass through it. Read more about these internal mechanisms in the article “ What happens in Kubernetes when kubectl run starts? ".



System audit is an interesting feature in Kubernetes, which is turned off by default. It allows you to log all calls to the Kubernetes API. As you can easily guess, through this API all actions related to monitoring and changing the state of the cluster are performed. A good description of its features can (as usual) be found in the official K8s documentation . Next, I will try to present the topic in a simpler language.



So, to enable auditing , we need to pass three required parameters to the container in the api-server, which are described in more detail below:





In addition to these three necessary parameters, there are many additional settings related to auditing: from log rotation to webhook descriptions. Example log rotation parameters:





But we will not dwell on them in more detail - you can find all the details in the documentation for kube-apiserver .



As already mentioned, all parameters are set in the manifest with the api-server configuration (by default /etc/kubernetes/manifests/kube-apiserver.yaml



), in the command



section. Let's go back to the 3 required parameters and analyze them:



  1. audit-policy-file



    - path to the YAML file with the description of the audit policy. We will return to its contents, but for now I note that the file should be accessible for reading by the api-server process. Therefore, it is necessary to mount it inside the container, for which you can add the following code to the appropriate sections of the config: 



      volumeMounts: - mountPath: /etc/kubernetes/policies name: policies readOnly: true volumes: - hostPath: path: /etc/kubernetes/policies type: DirectoryOrCreate name: policies
  2. audit-log-path



    - path to the log file. The path should also be accessible to the api-server process, therefore, we similarly describe its mounting: 



      volumeMounts: - mountPath: /var/log/kube-audit name: logs readOnly: false volumes: - hostPath: path: /var/log/kube-audit type: DirectoryOrCreate name: logs
  3. audit-log-format



    - format of the audit log. By default, this is json



    , but the legacy text format is also available.


Audit policy



Now about the mentioned file with the description of the logging policy. The first concept of audit policy is level



, the level of logging . They are as follows:





The last two levels ( Request



and RequestResponse



) do not log requests that did not access resources (calls to so-called non-resources urls).



Also, all requests go through several stages :





You can use omitStages



to skip any stages.



In the policy file, we can describe several sections with different levels of logging. The first matching rule found in the policy description will apply.



The kubelet daemon monitors the manifest change with the api-server configuration, and if any is detected, it restarts the container with api-server. But there is an important detail: they will ignore changes to the policy file . After making changes to the policy file, you will need to restart api-server manually. Since api-server is running as a static pod , the kubectl delete



command kubectl delete



not restart it. You have to manually make docker stop



on kube-masters where the audit policy has been changed: 



 docker stop $(docker ps | grep k8s_kube-apiserver | awk '{print $1}')


When you enable auditing, it is important to remember that the load on kube-apiserver increases . In particular, memory consumption for storing the context of requests is increasing. Logging starts only after sending the response header. Also, the load depends on the configuration of the audit policy.



Policy Examples



Let's analyze the structure of policy files using examples.



Here is a simple policy



file to log everything at the Metadata



level: 



 apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata


You can specify a list of users ( Users



and ServiceAccounts



) and user groups in the policy. For example, this is how we will ignore system users, but log everything else at the Request



level: 



 apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: None userGroups: - "system:serviceaccounts" - "system:nodes" users: - "system:anonymous" - "system:apiserver" - "system:kube-controller-manager" - "system:kube-scheduler" - level: Request


It is also possible to describe the target:





Note! Resources and resource groups (API groups, i.e. apiGroups), as well as their versions installed in the cluster, can be obtained using the commands: 



 kubectl api-resources kubectl api-versions


The following audit policy is provided as a demonstration of best practices in the Alibaba Cloud documentation : 



 apiVersion: audit.k8s.io/v1beta1 kind: Policy #    RequestReceived omitStages: - "RequestReceived" rules: #   ,     : - level: None users: ["system:kube-proxy"] verbs: ["watch"] resources: - group: "" #  api group   ,    #   Kubernetes,  “core” resources: ["endpoints", "services"] - level: None users: ["system:unsecured"] namespaces: ["kube-system"] verbs: ["get"] resources: - group: "" # core resources: ["configmaps"] - level: None users: ["kubelet"] verbs: ["get"] resources: - group: "" # core resources: ["nodes"] - level: None userGroups: ["system:nodes"] verbs: ["get"] resources: - group: "" # core resources: ["nodes"] - level: None users: - system:kube-controller-manager - system:kube-scheduler - system:serviceaccount:kube-system:endpoint-controller verbs: ["get", "update"] namespaces: ["kube-system"] resources: - group: "" # core resources: ["endpoints"] - level: None users: ["system:apiserver"] verbs: ["get"] resources: - group: "" # core resources: ["namespaces"] #     read-only URLs: - level: None nonResourceURLs: - /healthz* - /version - /swagger* #   ,     “”: - level: None resources: - group: "" # core resources: ["events"] #   Secret, ConfigMap  TokenReview    , #         - level: Metadata resources: - group: "" # core resources: ["secrets", "configmaps"] - group: authentication.k8s.io resources: ["tokenreviews"] #   get, list  watch   ;    - level: Request verbs: ["get", "list", "watch"] resources: - group: "" # core - group: "admissionregistration.k8s.io" - group: "apps" - group: "authentication.k8s.io" - group: "authorization.k8s.io" - group: "autoscaling" - group: "batch" - group: "certificates.k8s.io" - group: "extensions" - group: "networking.k8s.io" - group: "policy" - group: "rbac.authorization.k8s.io" - group: "settings.k8s.io" - group: "storage.k8s.io" #        API - level: RequestResponse resources: - group: "" # core - group: "admissionregistration.k8s.io" - group: "apps" - group: "authentication.k8s.io" - group: "authorization.k8s.io" - group: "autoscaling" - group: "batch" - group: "certificates.k8s.io" - group: "extensions" - group: "networking.k8s.io" - group: "policy" - group: "rbac.authorization.k8s.io" - group: "settings.k8s.io" - group: "storage.k8s.io" #         - level: Metadata




Another good audit policy example is the profile used in GCE .



For quick response to audit events, it is possible to describe a webhook . This issue is disclosed in the official documentation , I will leave it outside the scope of this article.



Summary



The article provides an overview of the basic security mechanisms in Kubernetes clusters that allow you to create personalized user accounts, share their rights, and register their actions. I hope it is useful to those who are faced with such questions in theory or already in practice. I also recommend that you read the list of other materials on the topic of security in Kubernetes, which is listed in the "PS" - perhaps among them you will find the necessary details on issues that are relevant to you.



PS



Read also in our blog:





More articles:


All Articles