I am trying to deploy Metricbeat on a K8s cluster set up using kubeadm (v1.14), and I have encountered several issues.
- The documentation lists a manifest file that tries to collect metrics from the kubelet read-only port (10255), but that port is no longer open (I believe kubeadm is defaulting to disabling the port from even before v1.12?).
This is mentioned here , but I was unable to find the corresponding github issue.
Maybe this could be added to the metricbeat documentation (including a sample manifest with a working config using the SSL port 10250?)
- When trying to switch to the SSL port, I encountered another issue - the kubelet certificate does not validate when using
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt.
This was discussed here, but no solution seems to have been reached (other than completely disabling SSL validation) .
I believe this is because of an open issue with kubeadm - when setting up the cluster using kubeadm, the kubelet uses a self-signed certificate instead of one signed by the cluster CA. See here , and in particular this comment for more details.
And indeed, this is what seems to be happening in my case. The kubelet certificate is self-signed (CN=node_name, instead of CN=kubernetes).
Server certificate
subject=/CN=kube-b-00-04@1557566648
issuer=/CN=kube-b-00-04-ca@1557566647
The only workaround I could find was to also disable SSL verification. ( ssl.verification_mode: none ).
In this case, since the issue is on kubeadm's side, I doubt there is anything that can be done on the metricbeat side. But maybe this could also be added to the docs, in case other people are facing the same issue.