Kubernetes Module [Kubelet SSL]


(Rodrigo Porto) #1

Hi everyone,

I have installed Kubernetes v1.11.2 by Kubeadm. I have configured Metricbeat (on Kubernetes) in order to get some metrics from Kubelet, however, I am not able to run it.

I suppose that I have to use the certificates which were created in /etc/kubernetes/pki.

ls /etc/kubernetes/pki/
apiserver.crt              apiserver-etcd-client.key  apiserver-kubelet-client.crt  ca.crt  etcd                front-proxy-ca.key      front-proxy-client.key  sa.pub
apiserver-etcd-client.crt  apiserver.key              apiserver-kubelet-client.key  ca.key  front-proxy-ca.crt  front-proxy-client.crt  sa.key

An example of Metricbeat's deployment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      hosts: ["https://localhost:10250"]
      ssl.certificate_authorities: ["/etc/certs/ca.crt"]
      ssl.certificate: "/etc/certs/apiserver-kubelet-client.crt"
      ssl.key: "/etc/certs/apiserver-kubelet-client.key"
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat
spec:
  template:
    metadata:
      labels:
        k8s-app: metricbeat
    spec:
      hostNetwork: true
      containers:
      - name: metricbeat
        image: docker.elastic.co/beats/metricbeat:6.4.0
        args: [ "-c", "/etc/metricbeat.yml", "-e", ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
        - name: kubelet
          mountPath: /etc/certs
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-config
      - name: modules
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-modules
      - name: kubelet
        secret:
         secretName: kubelet

On Elasticsearch, Metricbeat creates an index, nevertheless there is not any information about Kubernetes metrics.

error making http request: Get https://localhost:10250/stats/summary: x509: certificate is valid for k8s-test, not localhost

Thanks in advance,

Regards


(Carlos Pérez Aradros) #2

Hi @RdrgPorto,

Thank you for your feedback! We recently implemented bearer token file auth for this specific use case, here you have an example of how the kubernetes module conf should look:

Let us know how it goes if you test it! We plan to switch our example manifests to this on the next release, we may need to backport it to 6.4 as it's mandatory already in latest k8s version.


(Rodrigo Porto) #3

Hi @exekias,

I have just modified my Metricbeat's configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      hosts: ["https://localhost:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.certificate_authorities:
       - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

I get the same error:

error making http request: Get https://localhost:10250/stats/summary: x509: certificate is valid for k8s-test, not localhost

However, if I try to get information by curl:

~$ sudo curl https://localhost:10250/stats/summary -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key                      
{
  "node": {
   "nodeName": "k8s-test",
   "systemContainers": [
    {
     "name": "pods",
     "startTime": "2018-09-10T07:00:03Z",
     "cpu": {
      "time": "2018-09-10T07:06:06Z",
      "usageNanoCores": 171793678,
      "usageCoreNanoSeconds": 99047530865217
     },
     "memory": {
      "time": "2018-09-10T07:06:06Z",
      "availableBytes": 8376406016,
      "usageBytes": 3201318912,
      "workingSetBytes": 3165560832,
      "rssBytes": 3020820480,
      "pageFaults": 0,
      "majorPageFaults": 0
     },
......

Thanks in advance,

Regards


(Carlos Pérez Aradros) #4

Hi @RdrgPorto, it seems you would need to use k8s-test instead of localhost. I'm guessing that's the hostname of the machine, so probably hosts: ["https://${HOSTNAME}:10250"] is better


(Rodrigo Porto) #5

Hi @exekias,

I get the following error:

error making http request: Get https://k8s-test:10250/stats/summary: x509: certificate signed by unknown authority

Thanks in advance,

Regards


(Carlos Pérez Aradros) #6

That's interesting, can you try to list the available subjects from that CA file? you would need to go into the container and do something like:

awk -v cmd='openssl x509 -noout -subject' '
    /BEGIN/{close(cmd)};{print | cmd}' < /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

probably openssl is not installed in the Metricbeat container, you will need to install it with yum


(Rodrigo Porto) #7

Hi @exekias,

It returns:

subject= /CN=kubernetes

As far as I know, when one deploys Kubernetes by Kubeadm (kubeadm init):

[certificates] apiserver serving cert is signed for DNS names [k8s-test kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 my_ip]
[certificates] etcd/server serving cert is signed for DNS names [k8s-test localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-test localhost] and IPs [my_ip 127.0.0.1 ::1]

Moreover, there are only two services created on Kubernetes:

$ kubectl get svc --all-namespaces
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
default       kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP         23m
kube-system   kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   23m

If I use kubernetes as CNAME, I get the following error:

error making http request: Get https://kubernetes:10250/stats/summary: lookup kubernetes on 10.96.0.10:53: read udp 10.32.0.5:34401-\u003e10.96.0.10:53: i/o timeout

Thanks in advance,

Regards


(Carlos Pérez Aradros) #8

Problem is that we want to talk to kubelet, kubernetes hostname will resolve to the API server. I'm wondering why k8s-test is not resolving to 127.0.0.1, isn't that the name of the node?


(Rodrigo Porto) #9

Hi @exekias

/etc/hosts file:

127.0.0.1 k8s-test localhost
my_ip     k8s-test  
# The following lines are desirable for IPv6 capable hosts
::1       ip6-localhost ip6-loopback
ff02::1   ip6-allnodes
ff02::2   ip6-allrouters

Thanks in advance,

Regards


(Carlos Pérez Aradros) #10

So you confirm that setting hosts: ["https://k8s-test:10250"] gives you a CA error?

Of course this can be avoided by setting ssl.verification_mode: none, but that would be considered unsafe.

I will need to test this myself, as It's supposed to be working

Br,


(Rodrigo Porto) #11

Hi @exekias,

I don't know very well where the issue might be. Possibly, Kubeadm has changed something regarding to Kubelet.

By setting ssl.verification_mode: none, it works.

Thank you very much for your patience,

Have a nice weekend,

Regards :vulcan_salute:


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.