Hi, I have a 3 master, multiple data node architecture. Metricbeat's elasticsearch-xpack module is configured to ship elasticsearch metrics to a separate monitoring cluster. However, I see only node_stats are sent and no cluster_stats. because fo which, the Kibana monitoring page doesn't show any data.
Below is the elasticsearch-xpack.yml
- module: elasticsearch
metricsets:
- ccr
- cluster_stats
- enrich
- index
- index_recovery
- index_summary
- ml_job
- node_stats
- shard
period: 10s
hosts: ["https://{{ELASTICSEARCH_URL}}:9200"]
username: "elastic"
password: "{{MONITORING_USER_PASSWORD}}"
xpack.enabled: true
metricbeat.yml
#========================== Modules configuration ============================
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
#============================== Kibana =====================================
setup.kibana:
host: "{{MONITORING_KIBANA_URL}}:5601"
protocol: "https"
username: {{USERNAME}}
password: {{PASSWORD}}
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["{{MONITORING_ELASTICSEARCH_URL}}:9200"]
protocol: "https"
username: {{USERNAME}}
password: {{PASSWORD}}
#================================ Processors =====================================
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
#============================== X-Pack Monitoring ===============================
monitoring.enabled: true
monitoring.elasticsearch:
hosts: ["{{MONITORING_ELASTICSEARCH_URL}}:9200"]
protocol: "https"
username: "elastic"
password: {{PASSWORD}}
I have tested the config and it all looks fine. I have tested output
./metricbeat test output
elasticsearch: https://xxxxx:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: x.x.x.x, x.x.x.x
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 7.7.0
./metricbeat test modules
The above image says, cluster_stats events are not captured. However, node_stats events are.
From the below debug logs, it looks like metricbeat is detecting the node as a non-master node. However, the node is a master node.
Not sure, why.
On further debugging, found out that the node name returned from the _cluster/state/master_node and _nodes/_local/name are not same. And this is very strange.
Could anyone please help on this?