Kubernetes metricbeat module - how to setup?

[asp]

Hi,

I am quite new to kubernetes and want to run metricbeat in kubernetes, so that I can see whats going on in kubernetes in my elastic stack.
Kubernetes is running on bare metal, so there is no cloud provider involved.

Just for your information. In my case metricbeat is shipping it's data to redis, then logstash pulls it and pushes to elasticsearch.

I've deployed kubernetes like written here: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html

Hopefully I installed kube-state-metrics correctly. At least the kubernetes service is answering.

If I check kibana, I can find following events with error messages:

# event.dataset: kubernetes.node:  
error making http request: Get http://localhost:10255/stats/summary: dial tcp 127.0.0.1:10255: connect: connection refused

# event.dataset: kubernetes.container: 
error doing HTTP request to fetch 'container' Metricset data: error making http request: Get http://localhost:10255/stats/summary: dial tcp 127.0.0.1:10255: connect: connection refused

Did I miss anything to setup on kubernetes to get these metrics?
Any help is appreciated.

Thanks, Andreas

[asp]

I digged a bit deeper. Looks like metricbeat is trying to access the read-only-port of kubelet.
Seems it is disabled by default.
If I add --read-only-port to kubelet the port is available and can be fetched by metricbeat.

BUT, as I understand this had been changed in kubernetes for security reasons. So I don't want to weaken security getting metricbeat data.

As I understand, there is port 10250 with enabled TLS and client authentication. As described here
https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html
there are some parameters:

 #ssl.certificate_authorities:
 #  - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
 #ssl.certificate: "/etc/pki/client/cert.pem"
 #ssl.key: "/etc/pki/client/cert.key"

Which can be set for client authentication. But I am quite unfamiliar with these client certificates and I am thankful for further help:

• as the daemonset is using dynamic names, the hostname of the metricbeat pod is changing with each starting of the pod. With client certificates kubelet should check if certificate matches the pod's host name. How do I create client certificate dynamically for the pod that kubelet is trusting?
  Can you point out the workflow or provide any tutorials?

Thanks, Andreas