Hi guys,
I am trying to setup Stack monitoring using metricbeat to move away from legacy collection. We run on v7.11.2 and RHEL7 with below setup:
node 1: Elasticsearch, logstash, kibana
node 2: Elasticsearch
node 3: Elasticsearch
I installed metricbeat on node1 and enabled Elasticsearch-xpack module. Since then I am getting a combination of timeout errors:
Nov 26 13:22:36 <ip> metricbeat: 2021-11-26T13:22:36.878Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.ml_job: error determining if connected Elasticsearch node is master: error making http request: Get "https://node-1-IP:9201/_nodes/_local/nodes": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Nov 26 13:22:40 <ip> metricbeat: 2021-11-26T13:22:40.758Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.enrich: error determining if connected Elasticsearch node is master: error making http request: Get "https://node-1-IP:9201/_cluster/state/master_node": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Nov 26 13:15:29 <ip> metricbeat: 2021-11-26T13:15:29.473Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get "https://node-1-IP:9201/_nodes/_local/nodes": read tcp node-1-IP:37438->node-1-IP:9201: i/o timeout
essentially its timing out connecting to ES on the same node (which is being monitored). Below is the configuration:
metricbeat.yml
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
setup.kibana:
output.elasticsearch:
hosts: ["https://elkhost:9201"]
username: <user>
password: <pass>
ssl.enabled: true
# ssl.certificate_authorities: ["/etc/metricbeat/ssl/AWS-ES-CLUSTER-CA.pem"]
ssl.certificate: "/etc/metricbeat/ssl/ES-DEV-NODE-1.cer"
ssl.key: "/etc/metricbeat/ssl/ES-DEV-NODE-1.key"
ssl.verification_mode: none
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
and Elasticsearch-xpack.yml
- module: elasticsearch
xpack.enabled: true
period: 10s
hosts: ["https://eshost:9201/"]
username: <user>
password: <pass>
ssl.certificate: "/etc/metricbeat/ssl/ES-DEV-NODE-1.cer"
ssl.key: "/etc/metricbeat/ssl/ES-DEV-NODE-1.key"
ssl.verification_mode: none
I checked other posts in the forum, they all seem to be connecting to other nodes or monitoring servers other than ES.
Can anyone please suggest the fix for this ?
Thank you