I have a similar question tothe one posted by souravsahoo "Cluster_stats not sent to monitoring cluster" but no answer was posted. I am having something similar to where I have installed metricbeats on a production cluster and configured them to send events to a monitoring cluster. Yet I can't see the cluster info on the monitor cluster. The monitor cluster was seeing it's nodes but I did not enable monitoring on it since I have a basic license and only want to monitor the production cluster with the monitor node.
If I look in the logs from a datanode of the prod cluster I am not sure if I am seeing cluster type events, yet the beats elasticsearch-xpack.yml is enabled.
below is a sample of the logs:
Blockquote
metricbeat[28475]: 2021-09-29T18:07:58.617Z#011INFO#011[monitoring]#011log/log.go:144#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":3823314659}},"memory":{"mem":{"usage":{"bytes":282624}}}},"cpu":{"system":{"ticks":5579930,"time":{"ms":2268}},"total":{"ticks":9604270,"time":{"ms":3819},"value":9604270},"user":{"ticks":4024340,"time":{"ms":1551}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":20},"info":{"ephemeral_id":"17a6c8f3-","uptime":{"ms":71370084}},"memstats":{"gc_next":51721136,"memory_alloc":33304768,"memory_total":741217219480,"rss":112447488},"runtime":{"goroutines":118}},"libbeat":{"config":{"module":{"running":4}},"output":{"events":{"acked":59,"active":0,"batches":3,"total":59},"read":{"bytes":1520},"write":{"bytes":138576}},"pipeline":{"clients":19,"events":{"active":0,"published":59,"total":59},"queue":{"acked":59}}},"metricbeat":{"elasticsearch":{"node_stats":{"events":3,"success":3}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":5,"success":5},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":12,"success":12},"process":{"events":23,"success":23},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":5.27,"15":4.62,"5":4.51,"norm":{"1":0.1647,"15":0.1444,"5":0.1409}}}}}}
metricbeat[28475]: 2021-09-29T18:08:28.616Z#011INFO#011[monitoring]#011log/log.go:144#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":4116302266}},"memory":{"mem":{"usage":{"bytes":-139264}}}},"cpu":{"system":{"ticks":5582390,"time":{"ms":2451}},"total":{"ticks":9608400,"time":{"ms":4117},"value":9608400},"user":{"ticks":4026010,"time":{"ms":1666}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":20},"info":{"ephemeral_id":"17a6c8f3-","uptime":{"ms":71400086}},"memstats":{"gc_next":51654240,"memory_alloc":45236112,"memory_total":741528914936,"rss":112459776},"runtime":{"goroutines":118}},"libbeat":{"config":{"module":{"running":4}},"output":{"events":{"acked":54,"active":0,"batches":3,"total":54},"read":{"bytes":1489},"write":{"bytes":130469}},"pipeline":{"clients":19,"events":{"active":0,"published":54,"total":54},"queue":{"acked":54}}},"metricbeat":{"elasticsearch":{"node_stats":{"events":3,"success":3}},"system":{"cpu":{"events":3,"success":3},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":12,"success":12},"process":{"events":24,"success":24},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":6.81,"15":4.76,"5":4.95,"norm":{"1":0.2128,"15":0.1488,"5":0.1547}}}}}}
metricbeat[28475]: 2021-09-29T18:08:58.617Z#011INFO#011[monitoring]#011log/log.go:144#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":4239587871}},"memory":{"mem":{"usage":{"bytes":204800}}}},"cpu":{"system":{"ticks":5584990,"time":{"ms":2601}},"total":{"ticks":9612640,"time":{"ms":4239},"value":9612640},"user":{"ticks":4027650,"time":{"ms":1638}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":20},"info":{"ephemeral_id":"17a6c8f3-","uptime":{"ms":71430084}},"memstats":{"gc_next":51792272,"memory_alloc":32825496,"memory_total":741840990088,"rss":112558080},"runtime":{"goroutines":118}},"libbeat":{"config":{"module":{"running":4}},"output":{"events":{"acked":59,"active":0,"batches":3,"total":59},"read":{"bytes":1522},"write":{"bytes":138581}},"pipeline":{"clients":19,"events":{"active":0,"published":59,"total":59},"queue":{"acked":59}}},"metricbeat":{"elasticsearch":{"node_stats":{"events":3,"success":3}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":5,"success":5},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":12,"success":12},"process":{"events":23,"success":23},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":5.13,"15":4.69,"5":4.72,"norm":{"1":0.1603,"15":0.1466,"5":0.1475}}}}}}
my elasticsearc-xpack.yml is basically the default with the username and password provided to access the node.
I have also ran the below to verify that it is enabled on the prod cluster and it is set to true.
Blockquote
PUT _cluster/settings
{
"persistent": {
"xpack.monitoring.collection.enabled": false
}
}
BlockquoteUnable to find the cluster in the selected time range. UUID: ev-JuQs_RxK_AZgBIPdfddA
I can also see the prod data node metrics via metrics.