Metricbeat stack monitoring : i/o timeout when connecting to the same node

[preetish_P (Preetish Pai)]

Hi guys,

I am trying to setup Stack monitoring using metricbeat to move away from legacy collection. We run on v7.11.2 and RHEL7 with below setup:

node 1: Elasticsearch, logstash, kibana
node 2: Elasticsearch
node 3: Elasticsearch

I installed metricbeat on node1 and enabled Elasticsearch-xpack module. Since then I am getting a combination of timeout errors:

Nov 26 13:22:36 <ip> metricbeat: 2021-11-26T13:22:36.878Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.ml_job: error determining if connected Elasticsearch node is master: error making http request: Get "https://node-1-IP:9201/_nodes/_local/nodes": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Nov 26 13:22:40 <ip> metricbeat: 2021-11-26T13:22:40.758Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.enrich: error determining if connected Elasticsearch node is master: error making http request: Get "https://node-1-IP:9201/_cluster/state/master_node": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Nov 26 13:15:29 <ip> metricbeat: 2021-11-26T13:15:29.473Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get "https://node-1-IP:9201/_nodes/_local/nodes": read tcp node-1-IP:37438->node-1-IP:9201: i/o timeout

essentially its timing out connecting to ES on the same node (which is being monitored). Below is the configuration:

metricbeat.yml

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
output.elasticsearch:
  hosts: ["https://elkhost:9201"]
  username: <user>
  password: <pass>
  ssl.enabled: true
#  ssl.certificate_authorities: ["/etc/metricbeat/ssl/AWS-ES-CLUSTER-CA.pem"]
  ssl.certificate: "/etc/metricbeat/ssl/ES-DEV-NODE-1.cer"
  ssl.key: "/etc/metricbeat/ssl/ES-DEV-NODE-1.key"
  ssl.verification_mode: none
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

and Elasticsearch-xpack.yml

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://eshost:9201/"]
  username: <user>
  password: <pass>
  ssl.certificate: "/etc/metricbeat/ssl/ES-DEV-NODE-1.cer"
  ssl.key: "/etc/metricbeat/ssl/ES-DEV-NODE-1.key"
  ssl.verification_mode: none

I checked other posts in the forum, they all seem to be connecting to other nodes or monitoring servers other than ES.

Can anyone please suggest the fix for this ?

Thank you