Elasticsearch node with Metricbeat not registering in Kibana

[Chris_Stone]

I've added a 4th identical ES node to my cluster, and for some reason Metricbeat is not reporting correctly to Kibana.

The only error from the Metricbeat logs is:

Apr  4 17:58:35 elk-5 metricbeat[8781]: 2022-04-04T10:58:35.629-0700#011ERROR#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.cluster_stats: error making http request: Get "https://elk-2.corp.XXXXX.com:9200/_cluster/stats": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

But looks like it connect OK via curl:

root@elk-5:~# curl -XGET -I https://elk-2.corp.XXXXX.com:9200/_cluster/stats
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="security" charset="UTF-8"
WWW-Authenticate: Bearer realm="security"
WWW-Authenticate: ApiKey
content-type: application/json; charset=UTF-8
content-length: 487

But this looks OK, no?

metricbeat[8781]: 2022-04-04T10:57:21.243-0700#011INFO#011[monitoring]#011log/log.go:184#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":-27754496}}}},"cpu":{"system":{"ticks":4908000,"time":{"ms":902}},"total":{"ticks":40826610,"time":{"ms":5814},"value":40826610},"user":{"ticks":35918610,"time":{"ms":4912}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":18},"info":{"ephemeral_id":"77f55a2e-2259-4474-989e-3eeafd732023","uptime":{"ms":246513107},"version":"7.17.2"},"memstats":{"gc_next":136159040,"memory_alloc":71992056,"memory_total":6069660873112,"rss":291917824},"runtime":{"goroutines":84}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":9587,"active":9,"batches":344,"duplicates":7424,"total":17011},"read":{"bytes":5996379},"write":{"bytes":43746814}},"pipeline":{"clients":9,"events":{"active":0,"published":17011,"total":17011},"queue":{"acked":17011}}},"metricbeat":{"elasticsearch":{"cluster_stats":{"events":4,"success":4},"enrich":{"events":9,"success":9},"index":{"events":5853,"success":5853},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":3,"success":3},"shard":{"events":11136,"success":11136}}},"system":{"load":{"1":0.74,"15":0.43,"5":0.66,"norm":{"1":0.0925,"15":0.0538,"5":0.0825}}}}}}

The node shows in Kibana as in the screenshot below:

[Chris_Stone]

Any ideas about this? Trying to find other things to try...

[zx8086]

All metricbeat.yml files are identical ?

Other than metricbeat, the node is healthy ?

[Chris_Stone]

Thanks! Yeah, they were identical, and that was the problem The host value in /etc/metricbeat/modules.d/elasticsearch-xpack.yml was set to the original host and not the localhost. I'll just change them all to localhost. In any case, it's working now. Thanks again!

[zx8086]

@Chris_Stone that is what i meant, the configuration should be identical to the working ones, as usually you don't need to make them unique when using the default setup.