Metricbeat not reporting on port 10255 anymore using latest kubernetes, how to use the secured port 10250?

[pastorsx (simon)]

Hi all,

I've been using metricbeat through its helm chart to install in a kubernetes cluster. It used to work by going though port 10255 which was a non-secured read only port.. but it seems latest releases of kubernetes (installed through kubeadm) deprecated it and we are now forced to use a secure port...

I just cannot make it work... can anyone tell me what are the valid instructions for that:

previous config:
kubernetes:
enabled: true
config:
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
hosts: ["http://localhost:10255"]
add_metadata: true
in_cluster: true
processors:
- add_cloud_metadata:
- add_kubernetes_metadata:
in_cluster: true

new config that is not working:
kubernetes:
enabled: true
config:
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
hosts: ["https://${HOSTNAME}:10250"]
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.certificate_authorities:
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
add_metadata: true
in_cluster: true
processors:
- add_cloud_metadata:
- add_kubernetes_metadata:
in_cluster: true

When using this config, I get errors that the "x509: certificate was signed by unknown authority" error

When I use the curl comment on the host, I can have a good response:
sudo curl https://localhost:10250/stats/summary -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key

Anyone was able to get around it...

NOTE: helm chart uses a daemonset rather than a deployment.. I don't know if that may have an impact at all.. but I thought it was worth mentioning.

thanks

[exekias (Carlos Pérez Aradros)]

Hi @pastorsx,

You can use bearer_token_file like shown here: https://github.com/elastic/beats/blob/master/metricbeat/modules.d/kubernetes.yml.disabled#L13-L16

i think it's time to switch to this approach by default, as most kubernetes versions out there will support it already. Would you mind creating a new issue for that? https://github.com/elastic/beats/issues/new

Br,

[pastorsx (simon)]

Thanks, I will raise an issue.... but I tried this approach as it shows above... the file service-ca.crt does not exist... although ca.crt does, so I have been using it... but when I do, I get the "x509: certificate was signed by unknown authority" here as a result of the requests so I must be missing something else... any idea?

thanks again for your reply...

[exekias (Carlos Pérez Aradros)]

Yes, probably you have to change localhost to the name of the node. In our default manifests we added the NODE_NAME env var for this. So if you have it reported, just switch to ${NODE_NAME}:10250.

[pastorsx (simon)]

yeah.. thanks, I tried that too... that's what I have, but still no luck (get the same x509 error):

kubernetes:
enabled: true
config:

• module: kubernetes
  metricsets:
• node
• system
• pod
• container
• volume
  period: 10s
  hosts: ["https://${HOSTNAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.certificate_authorities:
• /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  add_metadata: true
  in_cluster: true

[Artur_Becker]

I have a similar problem. I am using Metricbeat version 6.4 and Kubernetes version v1.12.1. If I run the following Curl command, I get data.

curl https://localhost:10250/stats/summary -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key

I use the same configuration as "pastorsx simon", but Metricbeat will not start because /var/run/secrets/ does not exist on the system. Is this path created when installing Kubernetes?

Greetings