In my last article, I showed how to develop the KubeLogExporter, a tool that collects log data from a set of pods. So far the exporter relies on full access to the Cluster by using the local .kubeconfig
file. If we want the exporter to run as a cron job inside the cluster, it needs to have suitable access rights. Originally I wanted to just write about the cron job implementation, but found that investigating how Kubernetes access rights work to be very educational. That’s why it became the article that you are reading now.
This article originally appeared at my blog.
Essential Kubernetes Concepts
When you run a Pod inside a Kubernetes cluster, several default configurations and security aspects are already made by default. To determine the access right to the rich Kubernetes API, the essential resources are ServiceAccount
, Role
and RoleBindings
.
Let’s understand these concepts by considering how the cron job to read pod logs works. When the job runs, it needs read access to namespaces and pods. This access is defined in a Role
or ClusterRole
. The Role
is limited to one namespace only, so we will use the ClusterRole
. When a pod is created, it is given the default system account and the default system account token to access the K8S API. However, this account does not have the required access rights, so we need to define a custom ServiceAccount
. The final piece is the RoleBinding
or ClusterRoleBinding
: It connects the ClusterRole
with the ServiceAccount
.
K8S API: Direct Access
To see how those concepts are applied when working with Kubernetes, I followed this excellent article in which the API is accessed directly with curl
.
Let’s start simple by creating the api-explorer
pod that by writing the api-explorer-pod.yaml
file with the following content.
apiVersion: v1
kind: Pod
metadata:
name: api-explorer
spec:
containers:
- name: alpine
image: alpine
args: ['sleep', '3600']
Then we create the container and wait until it is started.
> kubectl create -f api-explorer-pod.yamlpod/api-explorer created
Then we jump into the container, and install the curl
package.
> kubectl api-explorer -it sh
> apk add curl
To access the Kubernetes API, we always need to send a valid security token. This token is stored inside the pod at the location /run/secrets/kubernetes.io/serviceaccount/token
. With this token, we can make the API request.
> TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)
> curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/default/pods/ --insecure
{
"kind": "Status",
"apiVersion": "v1",
"metadata": { },
"status": "Failure",
"message": "pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
"reason": "Forbidden",
"details": {
"kind": "pods"
},
"code": 403
}
But as we see, we cannot access the pod at all because the default service account does not have the correct access rights.
Defining a Custom Service Account
So we need to have a properly configured ServiceAccount
that grants us a token with which the Kubernetes API can be accessed.
Create the file pod-read-access-service-account.yaml
and put the ServiceAccount
definition on top. This resource is basically only metadata.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: log-exporter-sa
namespace: default
labels:
app: log-exporter
---
The next thing is the ClusterRole
definition. Inside its spec
block, we define which apiGroups
and resources
we want to access. The core API group is denoted by ""
, and under resources
we list pods
. Finally, the verbs
determine which action we want to apply to the resources: In our case, its reading and listing.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: log-exporter-cr
labels:
app: log-exporter
rules:
- apiGroups:
- ''
resources:
- pods
- pods/log
- namespaces
verbs:
- get
- list
---
Finally we create the RoleBinding
resource to combine the SeviceAccount
and the ClusterRole
.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: log-exporter-rb
roleRef:
kind: ClusterRole
name: log-exporter-cr
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: log-exporter-sa
namespace: default
---
Now we create all resources.
> kubectl create -f pod-read-access-service-account.yamlserviceaccount/log-exporter-sa created
clusterrole.rbac.authorization.k8s.io/log-exporter-cr created
rolebinding.rbac.authorization.k8s.io/log-exporter-rb created
Some more details: As you see, the ServiceAccount
is explicitly created in the default
namespace. Be careful about the ClusterRoleBinding
as it needs to reference this ServiceAccount
in its defined namespace as well, or it will not work correctly.
K8S API: Access with Custom Service Account
To use the newly created ServiceAccount
, we define that the pod uses the new role. Going back to the api-explorer-pod.yaml
file, we add the new configuration item spec.serviceAccountName
.
apiVersion: v1
kind: Pod
metadata:
name: api-explorer
spec:
serviceAccountName: log-exporter-sa
containers:
- name: alpine
image: alpine
args: ['sleep', '3600']
Back in the container, we grab the token to make the request — and it works!
curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/default/pods/ --insecure
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/default/pods/",
"resourceVersion": "320995"
},
"items": [
{
"metadata": {
"name": "api-explorer2",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/api-explorer2",
"uid": "343aaf7e-1be5-45da-aadb-e83ee329a7fd",
"resourceVersion": "320976",
"creationTimestamp": "2020-05-24T10:16:58Z"
},
...
Now as the final proof of concepts, let’s try to read logs from a different pod in a different namespace. We grab the coredns
pod from the kube-system
namespace.
kb get pods -n kube-systemNAME READY STATUS RESTARTS AGE
metrics-server-6d684c7b5-6ww29 1/1 Running 7 8d
coredns-d798c9dd-pdswq 1/1 Running 7 8d
The url to access this pod is composed as this: /api/v1/namespaces/{namespace}/pods/{name}/log
. So, we need the exact namespace and the exact pod name for this request to work. Get back into the api-explorer
pod and access the log files.
> curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/kube-system/pods/coredns-d798c9dd-pdswq/log --insecure[INFO] plugin/reload: Running configuration MD5 = 4665410bf21c8b272fcfd562c482cb82
______ ____ _ _______
/ ____/___ ________ / __ \/ | / / ___/ ~ CoreDNS-1.6.3
/ / / __ \/ ___/ _ \/ / / / |/ /\__ \ ~ linux/arm, go1.12.9
/ /___/ /_/ / / / __/ /_/ / /| /___/ /
\____/\____/_/ \___/_____/_/ |_//____/
We are happy to see that it works as intended.
Conclusion
In this article, we learned about the essential Kubernetes resources for enabling to access pods and their log files in any namespace. A ClusterRole
defines which resources and which actions on those resources should be given. These access rights are bound with a ClusterRoleBinding
to a ServiceAccount
. Then we use this ServiceAccount
to provide the access rights to a Pod
. We showed how the Kubernetes API can be access from within a pod by using the curl
command. In the next article we will see how to implement a cron job that uses this service account to export the log files.