Metricbeat not reporting on port 10255 anymore using latest kubernetes, how to use the secured port 10250?

Hi all,

I've been using metricbeat through its helm chart to install in a kubernetes cluster. It used to work by going though port 10255 which was a non-secured read only port.. but it seems latest releases of kubernetes (installed through kubeadm) deprecated it and we are now forced to use a secure port...

I just cannot make it work... can anyone tell me what are the valid instructions for that:

previous config:
kubernetes:
enabled: true
config:
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
hosts: ["http://localhost:10255"]
add_metadata: true
in_cluster: true
processors:
- add_cloud_metadata:
- add_kubernetes_metadata:
in_cluster: true

new config that is not working:
kubernetes:
enabled: true
config:
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
hosts: ["https://${HOSTNAME}:10250"]
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.certificate_authorities:
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
add_metadata: true
in_cluster: true
processors:
- add_cloud_metadata:
- add_kubernetes_metadata:
in_cluster: true

When using this config, I get errors that the "x509: certificate was signed by unknown authority" error

When I use the curl comment on the host, I can have a good response:
sudo curl https://localhost:10250/stats/summary -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key

Anyone was able to get around it...

NOTE: helm chart uses a daemonset rather than a deployment.. I don't know if that may have an impact at all.. but I thought it was worth mentioning.

thanks

Hi @pastorsx,

You can use bearer_token_file like shown here: https://github.com/elastic/beats/blob/master/metricbeat/modules.d/kubernetes.yml.disabled#L13-L16

i think it's time to switch to this approach by default, as most kubernetes versions out there will support it already. Would you mind creating a new issue for that? https://github.com/elastic/beats/issues/new

Br,

Thanks, I will raise an issue.... but I tried this approach as it shows above... the file service-ca.crt does not exist... although ca.crt does, so I have been using it... but when I do, I get the "x509: certificate was signed by unknown authority" here as a result of the requests so I must be missing something else... any idea?

thanks again for your reply...

Yes, probably you have to change localhost to the name of the node. In our default manifests we added the NODE_NAME env var for this. So if you have it reported, just switch to ${NODE_NAME}:10250.

yeah.. thanks, I tried that too... that's what I have, but still no luck (get the same x509 error):

kubernetes:
enabled: true
config:

  • module: kubernetes
    metricsets:
  • node
  • system
  • pod
  • container
  • volume
    period: 10s
    hosts: ["https://${HOSTNAME}:10250"]
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    ssl.certificate_authorities:
  • /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    add_metadata: true
    in_cluster: true

I have a similar problem. I am using Metricbeat version 6.4 and Kubernetes version v1.12.1. If I run the following Curl command, I get data.

curl https://localhost:10250/stats/summary -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key

I use the same configuration as "pastorsx simon", but Metricbeat will not start because /var/run/secrets/ does not exist on the system. Is this path created when installing Kubernetes?

Greetings

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.