# Kubernetes Module \[Kubelet SSL\]

**URL:** <https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [September 7, 2018, 8:21pm UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770 "2018-09-07T20:21:33Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [September 7, 2018, 8:21pm UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/1 "2018-09-07T20:21:33Z")

</div>

Hi everyone,

I have installed **Kubernetes v1.11.2** by **Kubeadm**. I have configured **Metricbeat** (on **Kubernetes** ) in order to get some metrics from **Kubelet** , however, I am not able to run it.

I suppose that I have to use the certificates which were created in **/etc/kubernetes/pki**.

```
ls /etc/kubernetes/pki/
apiserver.crt apiserver-etcd-client.key apiserver-kubelet-client.crt ca.crt etcd front-proxy-ca.key front-proxy-client.key sa.pub
apiserver-etcd-client.crt apiserver.key apiserver-kubelet-client.key ca.key front-proxy-ca.crt front-proxy-client.crt sa.key

```

An example of **Metricbeat's deployment** :

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      hosts: ["https://localhost:10250"]
      ssl.certificate_authorities: ["/etc/certs/ca.crt"]
      ssl.certificate: "/etc/certs/apiserver-kubelet-client.crt"
      ssl.key: "/etc/certs/apiserver-kubelet-client.key"
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat
spec:
  template:
    metadata:
      labels:
        k8s-app: metricbeat
    spec:
      hostNetwork: true
      containers:
      - name: metricbeat
        image: docker.elastic.co/beats/metricbeat:6.4.0
        args: ["-c", "/etc/metricbeat.yml", "-e",]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
        - name: kubelet
          mountPath: /etc/certs
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-config
      - name: modules
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-modules
      - name: kubelet
        secret:
         secretName: kubelet

```

On **Elasticsearch** , **Metricbeat** creates an index, nevertheless there is not any information about **Kubernetes metrics**.

> error making http request: Get [https://localhost:10250/stats/summary:](https://localhost:10250/stats/summary:) x509: certificate is valid for k8s-test, not localhost

Thanks in advance,

Regards

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [September 9, 2018, 8:21pm UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/2 "2018-09-09T20:21:02Z")

</div>

Hi @RdrgPorto,

Thank you for your feedback! We recently implemented bearer token file auth for this specific use case, here you have an example of how the `kubernetes` module conf should look:

> <https://github.com/elastic/beats/blob/master/metricbeat/modules.d/kubernetes.yml.disabled#L13-L16>

Let us know how it goes if you test it! We plan to switch our example manifests to this on the next release, we may need to backport it to 6.4 as it's mandatory already in latest k8s version.

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [September 10, 2018, 7:10am UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/3 "2018-09-10T07:10:27Z")

</div>

Hi @exekias,

I have just modified my **Metricbeat's configmap** :

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      hosts: ["https://localhost:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.certificate_authorities:
       - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

```

I get the same error:

> error making http request: Get [https://localhost:10250/stats/summary:](https://localhost:10250/stats/summary:) x509: certificate is valid for k8s-test, not localhost

However, if I try to get information by **curl** :

```
~$ sudo curl https://localhost:10250/stats/summary -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key                      
{
  "node": {
   "nodeName": "k8s-test",
   "systemContainers": [
    {
     "name": "pods",
     "startTime": "2018-09-10T07:00:03Z",
     "cpu": {
      "time": "2018-09-10T07:06:06Z",
      "usageNanoCores": 171793678,
      "usageCoreNanoSeconds": 99047530865217
     },
     "memory": {
      "time": "2018-09-10T07:06:06Z",
      "availableBytes": 8376406016,
      "usageBytes": 3201318912,
      "workingSetBytes": 3165560832,
      "rssBytes": 3020820480,
      "pageFaults": 0,
      "majorPageFaults": 0
     },
......

```

Thanks in advance,

Regards

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [September 10, 2018, 10:27am UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/4 "2018-09-10T10:27:23Z")

</div>

Hi @RdrgPorto, it seems you would need to use `k8s-test` instead of `localhost`. I'm guessing that's the hostname of the machine, so probably `hosts: ["https://${HOSTNAME}:10250"]` is better

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [September 10, 2018, 11:14am UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/5 "2018-09-10T11:14:46Z")

</div>

Hi @exekias,

I get the following error:

> error making http request: Get [https://k8s-test:10250/stats/summary:](https://k8s-test:10250/stats/summary:) x509: certificate signed by unknown authority

Thanks in advance,

Regards

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [September 10, 2018, 11:23am UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/6 "2018-09-10T11:23:18Z")

</div>

That's interesting, can you try to list the available subjects from that CA file? you would need to go into the container and do something like:

```auto
awk -v cmd='openssl x509 -noout -subject' '
    /BEGIN/{close(cmd)};{print | cmd}' < /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

```

probably openssl is not installed in the Metricbeat container, you will need to install it with yum

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [September 10, 2018, 3:20pm UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/7 "2018-09-10T15:20:59Z")

</div>

Hi @exekias,

It returns:

> subject= /CN=kubernetes

As far as I know, when one deploys **Kubernetes** by **Kubeadm** ( **kubeadm init** ):

```
[certificates] apiserver serving cert is signed for DNS names [k8s-test kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 my_ip]
[certificates] etcd/server serving cert is signed for DNS names [k8s-test localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-test localhost] and IPs [my_ip 127.0.0.1 ::1]

```

Moreover, there are only two services created on **Kubernetes** :

```
$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 23m

```

If I use **kubernetes** as CNAME, I get the following error:

> error making http request: Get [https://kubernetes:10250/stats/summary:](https://kubernetes:10250/stats/summary:) lookup kubernetes on 10.96.0.10:53: read udp 10.32.0.5:34401-\u003e10.96.0.10:53: i/o timeout

Thanks in advance,

Regards

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [September 11, 2018, 2:23pm UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/8 "2018-09-11T14:23:57Z")

</div>

Problem is that we want to talk to kubelet, `kubernetes` hostname will resolve to the API server. I'm wondering why `k8s-test` is not resolving to 127.0.0.1, isn't that the name of the node?

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [September 14, 2018, 6:06am UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/9 "2018-09-14T06:06:18Z")

</div>

Hi @exekias

**/etc/hosts** file:

```
127.0.0.1 k8s-test localhost
my_ip k8s-test  
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

```

Thanks in advance,

Regards

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [September 14, 2018, 12:02pm UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/10 "2018-09-14T12:02:46Z")

</div>

So you confirm that setting `hosts: ["https://k8s-test:10250"]` gives you a CA error?

Of course this can be avoided by setting `ssl.verification_mode: none`, but that would be considered unsafe.

I will need to test this myself, as It's supposed to be working

Br,

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [September 14, 2018, 1:02pm UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/11 "2018-09-14T13:02:37Z")

</div>

Hi @exekias,

I don't know very well where the issue might be. Possibly, **Kubeadm** has changed something regarding to **Kubelet**.

By setting `ssl.verification_mode: none`, it works.

Thank you very much for your patience,

Have a nice weekend,

Regards 🖖

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2018, 1:11pm UTC](https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/12 "2018-10-12T13:11:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.