Kubernetes Module [Kubelet SSL]

RdrgPorto · September 7, 2018, 8:21pm

Hi everyone,

I have installed Kubernetes v1.11.2 by Kubeadm. I have configured Metricbeat (on Kubernetes) in order to get some metrics from Kubelet, however, I am not able to run it.

I suppose that I have to use the certificates which were created in /etc/kubernetes/pki.

ls /etc/kubernetes/pki/
apiserver.crt              apiserver-etcd-client.key  apiserver-kubelet-client.crt  ca.crt  etcd                front-proxy-ca.key      front-proxy-client.key  sa.pub
apiserver-etcd-client.crt  apiserver.key              apiserver-kubelet-client.key  ca.key  front-proxy-ca.crt  front-proxy-client.crt  sa.key

An example of Metricbeat's deployment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      hosts: ["https://localhost:10250"]
      ssl.certificate_authorities: ["/etc/certs/ca.crt"]
      ssl.certificate: "/etc/certs/apiserver-kubelet-client.crt"
      ssl.key: "/etc/certs/apiserver-kubelet-client.key"
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat
spec:
  template:
    metadata:
      labels:
        k8s-app: metricbeat
    spec:
      hostNetwork: true
      containers:
      - name: metricbeat
        image: docker.elastic.co/beats/metricbeat:6.4.0
        args: [ "-c", "/etc/metricbeat.yml", "-e", ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
        - name: kubelet
          mountPath: /etc/certs
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-config
      - name: modules
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-modules
      - name: kubelet
        secret:
         secretName: kubelet

On Elasticsearch, Metricbeat creates an index, nevertheless there is not any information about Kubernetes metrics.

error making http request: Get https://localhost:10250/stats/summary: x509: certificate is valid for k8s-test, not localhost

Thanks in advance,

Regards

exekias · September 9, 2018, 8:21pm

Hi @RdrgPorto,

Thank you for your feedback! We recently implemented bearer token file auth for this specific use case, here you have an example of how the kubernetes module conf should look:

github.com

elastic/beats/blob/master/metricbeat/modules.d/kubernetes.yml.disabled#L13-L16


hosts: ["localhost:10250"]
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.certificate_authorities:
  - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

Let us know how it goes if you test it! We plan to switch our example manifests to this on the next release, we may need to backport it to 6.4 as it's mandatory already in latest k8s version.

RdrgPorto · September 10, 2018, 7:10am

Hi @exekias,

I have just modified my Metricbeat's configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      hosts: ["https://localhost:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.certificate_authorities:
       - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

I get the same error:

error making http request: Get https://localhost:10250/stats/summary: x509: certificate is valid for k8s-test, not localhost

However, if I try to get information by curl:

~$ sudo curl https://localhost:10250/stats/summary -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key                      
{
  "node": {
   "nodeName": "k8s-test",
   "systemContainers": [
    {
     "name": "pods",
     "startTime": "2018-09-10T07:00:03Z",
     "cpu": {
      "time": "2018-09-10T07:06:06Z",
      "usageNanoCores": 171793678,
      "usageCoreNanoSeconds": 99047530865217
     },
     "memory": {
      "time": "2018-09-10T07:06:06Z",
      "availableBytes": 8376406016,
      "usageBytes": 3201318912,
      "workingSetBytes": 3165560832,
      "rssBytes": 3020820480,
      "pageFaults": 0,
      "majorPageFaults": 0
     },
......

Thanks in advance,

Regards

exekias · September 10, 2018, 10:27am

Hi @RdrgPorto, it seems you would need to use k8s-test instead of localhost. I'm guessing that's the hostname of the machine, so probably hosts: ["https://${HOSTNAME}:10250"] is better

RdrgPorto · September 10, 2018, 11:14am

Hi @exekias,

I get the following error:

error making http request: Get https://k8s-test:10250/stats/summary: x509: certificate signed by unknown authority

Thanks in advance,

Regards

exekias · September 10, 2018, 11:23am

That's interesting, can you try to list the available subjects from that CA file? you would need to go into the container and do something like:

awk -v cmd='openssl x509 -noout -subject' '
    /BEGIN/{close(cmd)};{print | cmd}' < /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

probably openssl is not installed in the Metricbeat container, you will need to install it with yum

RdrgPorto · September 10, 2018, 3:20pm

Hi @exekias,

It returns:

subject= /CN=kubernetes

As far as I know, when one deploys Kubernetes by Kubeadm (kubeadm init):

[certificates] apiserver serving cert is signed for DNS names [k8s-test kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 my_ip]
[certificates] etcd/server serving cert is signed for DNS names [k8s-test localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-test localhost] and IPs [my_ip 127.0.0.1 ::1]

Moreover, there are only two services created on Kubernetes:

$ kubectl get svc --all-namespaces
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
default       kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP         23m
kube-system   kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   23m

If I use kubernetes as CNAME, I get the following error:

error making http request: Get https://kubernetes:10250/stats/summary: lookup kubernetes on 10.96.0.10:53: read udp 10.32.0.5:34401-\u003e10.96.0.10:53: i/o timeout

Thanks in advance,

Regards

exekias · September 11, 2018, 2:23pm

Problem is that we want to talk to kubelet, kubernetes hostname will resolve to the API server. I'm wondering why k8s-test is not resolving to 127.0.0.1, isn't that the name of the node?

RdrgPorto · September 14, 2018, 6:06am

Hi @exekias

/etc/hosts file:

127.0.0.1 k8s-test localhost
my_ip     k8s-test  
# The following lines are desirable for IPv6 capable hosts
::1       ip6-localhost ip6-loopback
ff02::1   ip6-allnodes
ff02::2   ip6-allrouters

Thanks in advance,

Regards

exekias · September 14, 2018, 12:02pm

So you confirm that setting hosts: ["https://k8s-test:10250"] gives you a CA error?

Of course this can be avoided by setting ssl.verification_mode: none, but that would be considered unsafe.

I will need to test this myself, as It's supposed to be working

Br,

RdrgPorto · September 14, 2018, 1:02pm

Hi @exekias,

I don't know very well where the issue might be. Possibly, Kubeadm has changed something regarding to Kubelet.

By setting ssl.verification_mode: none, it works.

Thank you very much for your patience,

Have a nice weekend,

Regards

system · October 12, 2018, 1:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
[solved] [kubernetes module] unable to enable ssl Beats metricbeat	7	2968	April 17, 2018
Metricbeat fails to communicate with kubelet Beats metricbeat	2	749	February 19, 2019
Metricbeat Kubernetes Module Beats metricbeat	10	3091	August 2, 2017
Kubernetes Module - Kubelet -unauthorized error Beats metricbeat	3	2002	March 29, 2019
Kubernetes metricbeat module - how to setup? Beats metricbeat	2	951	September 10, 2019

Kubernetes Module [Kubelet SSL]

Related topics