Remote Elasticsearch Cluster monitoring through Metricbeat not disabling metricset

Elastic Stack Beats
1

Hello,

Scratching my head on this one.

I'm setting up remote elasticsearch monitoring for a production cluster and sending metrics to a second elasticsearch cluster to handle observing the production cluster. Everything seems to be in order for node monitoring, but when setting up cluster monitoring I get a 403 when it tries to query the elasticsearch.enrich dataset, presumably because we do not have enrichment enabled. So I just decided to disable that metricset (and ml_job which I know we have turned off). However, metricbeat does not collect any of the metrics I list out, and instead it fails on exactly the metricsets I do not want enabled and nothing more besides the system module which is working as expected. I can independently verify access of the remote users used to access both elasticsearch clusters and they do work for other modules, including node stats for the node level metricbeat instances.

My metricbeat config for the remote cluster monitoring is here:

################### metricbeat Configuration #########################

fields:
  env: dev
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
metricbeat.modules:
- hosts:
  - https://production-cluster.example.com:443
  metricsets:
  - cluster_stats
  - index
  - index_recovery
  - index_summary
  - pending_tasks
  - shard
  #- ccr
  #- enrich
  #- ml_job
  module: elasticsearch
  password: redacted
  period: 10s
  scope: cluster
  username: remote_system_monitoring
  xpack.enabled: true
processors:
- add_host_metadata: null
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup.ilm.rollover_alias: metricbeat-dev-%{[agent.version]}
tags:
- dev



###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features


output:
  elasticsearch:
    hosts:
    - https://remote-cluster-load-balancer.example.com:443
    password: redacted
    protocol: https
    username: elastic


############################# Logging #########################################

logging:
  files:
    rotateeverybytes: 10485760
logging.level: info

And taking out the systems metrics, the logs are littered with just:

Feb 20 22:55:28 remote_monitoring_host metricbeat[3000]: 2021-02-20T22:55:26.056Z        INFO        module/wrapper.go:259        Error fetching data for metricset elasticsearch.ml_job: HTTP error 400 in : 400 Bad Request
Feb 20 22:55:30 remote_monitoring_host metricbeat[3000]: 2021-02-20T22:55:30.569Z        INFO        module/wrapper.go:259        Error fetching data for metricset elasticsearch.enrich: HTTP error 403 in : 403 Forbidden

I'm aware of this topic that was raised a few years ago, but I've checked, all the elasticsearch modules in modules.d are disabled.

My understanding was commenting out the metricsets in the module config ensured they were removed from that beat's monitoring profile.

Everything is running on CentOS 7 using the rpm packages. I use the Elastic ansible galaxy roles to configure Elasticsearch and Metricbeat. The monitored cluster is running on 7.8.0 (long story) and the remote monitoring cluster is running on 7.10.2.

Truly I am confused and could use some guidance.

Thank you,
Devin

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.