Kubernetes data collection error with metricbeat!

Hello all!
I am trying to get kubernetes describing data (kubernetes.) via metricbeat. For installation i used metricbeat-kubernetes.yaml template. I have installed locally single-node kubernetes via docker, and many-node kubernetes on specifical host. So, MAIN problem is that local kubernetes cluster sends all data, including kubernetes., system. and so on, but another cluster sends only system.* data. When i found error why kubernetes data is not sending: "error making http request: Get http://localhost:10255/stats/summary: dial tcp localhost:10255: getsockopt: connection refused". So i changed localhost:10255 to (InternalIIP(master node) which i found in kubernetes dashboard):10255. In result, from local kubenetes cluster data about is still sending, but external cluster shows same error about "connection refused".
Can someone help with this problem?

Problem connected with this code part (HOSTS) ==>

kubernetes.yml: |-
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
hosts: ["localhost / IntrernalIP(master node) <== tried both:10255"]

External Kubernetes cluster (master node/all nodes) response:
Screenshot%20(186)

Local Kubernetes cluster (master node) response:

Hi @b22565875b81d856c5b0,

In localhost:10255 it is expected to find the kubelet read-only API, not to be mistaken by the node master or the apiserver. This kubelet API allows to collect info and metrics of the node where this kubelet is running. It should be available on all nodes of your cluster, you can check it by running curl localhost:10255 on any node, if it is not available there please check your kubelets status and configuration.

If kubelet is running, and it is reachable on localhost:10255, then there must be some connectivity problem between the metricbeat pod and the host network, please check that the metricbeat pods started by the daemon set on every node include the setting hostNetwork: true.

Thanks for response! I found that curl localhost:10255 is not available on my nodes. So i checked netstat and found that port 10255 dont exist at all. :frowning: How could i add needed port?

Kubelet config file:
Screenshot%20(190)
NetStat output:

What version of kubernetes are you using?

1.10.3.

{
"clientVersion": {
"major": "1",
"minor": "10",
"gitVersion": "v1.10.3",
"gitCommit": "2bba0127d85d5a46ab4b778548be28623b32d0b0",
"gitTreeState": "clean",
"buildDate": "2018-05-21T09:17:39Z",
"goVersion": "go1.9.3",
"compiler": "gc",
"platform": "linux/amd64"
},
"serverVersion": {
"major": "1",
"minor": "10",
"gitVersion": "v1.10.3",
"gitCommit": "2bba0127d85d5a46ab4b778548be28623b32d0b0",
"gitTreeState": "clean",
"buildDate": "2018-05-21T09:05:37Z",
"goVersion": "go1.9.3",
"compiler": "gc",
"platform": "linux/amd64"
}
}

It should be there then, kubelet opens this port unless it is modified or disabled with the --read-only-port flag. Can you share also the arguments you are running kubelet with?

I have found, that --read-only-port has been deprecated. Do you know how could i get it available?

Or how to get same information, if if would use 10250 (secure port) instead localhost:10255/stats/summary?

up...

Umm, yes, it seems that the flag is deprecated, but it could still be in use. This port is open by default, I have just checked in two different 1.10 clusters. Did you check if this flag was being used (ps aux | grep kubelet)?

How are you configuring your cluster? It seems that for example kubeadm disables the read-only port by default.

Entering "ps aux | grep kubelet" i have found that read-only-port is defined as 0.

How and which config file should i change to get available this port? etc/kubernetes/kubelet.conf?

It depends on what is adding this flag :slight_smile: This is why I asked about how you are managing the configuration of your cluster.

Honestly, i am newbie in kubernetes, so it is possible what i something can not understand, but when i check kubeadm, it shows what "not found". This cluster configured by my colegue from Ukraine, so many things about cluster i don`t know. :frowning:Screenshot%20(199)_LI

You'd need to check how kubelet is started, but probably better to wait for your colleague :slight_smile:

In the meantime if you want you can try to configure ssl to use the secure port.