Hello,
Scratching my head on this one.
I'm setting up remote elasticsearch monitoring for a production cluster and sending metrics to a second elasticsearch cluster to handle observing the production cluster. Everything seems to be in order for node monitoring, but when setting up cluster monitoring I get a 403 when it tries to query the elasticsearch.enrich dataset, presumably because we do not have enrichment enabled. So I just decided to disable that metricset (and ml_job which I know we have turned off). However, metricbeat does not collect any of the metrics I list out, and instead it fails on exactly the metricsets I do not want enabled and nothing more besides the system module which is working as expected. I can independently verify access of the remote users used to access both elasticsearch clusters and they do work for other modules, including node stats for the node level metricbeat instances.
My metricbeat config for the remote cluster monitoring is here:
################### metricbeat Configuration #########################
fields:
env: dev
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
metricbeat.modules:
- hosts:
- https://production-cluster.example.com:443
metricsets:
- cluster_stats
- index
- index_recovery
- index_summary
- pending_tasks
- shard
#- ccr
#- enrich
#- ml_job
module: elasticsearch
password: redacted
period: 10s
scope: cluster
username: remote_system_monitoring
xpack.enabled: true
processors:
- add_host_metadata: null
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup.ilm.rollover_alias: metricbeat-dev-%{[agent.version]}
tags:
- dev
###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features
output:
elasticsearch:
hosts:
- https://remote-cluster-load-balancer.example.com:443
password: redacted
protocol: https
username: elastic
############################# Logging #########################################
logging:
files:
rotateeverybytes: 10485760
logging.level: info
And taking out the systems metrics, the logs are littered with just:
Feb 20 22:55:28 remote_monitoring_host metricbeat[3000]: 2021-02-20T22:55:26.056Z INFO module/wrapper.go:259 Error fetching data for metricset elasticsearch.ml_job: HTTP error 400 in : 400 Bad Request
Feb 20 22:55:30 remote_monitoring_host metricbeat[3000]: 2021-02-20T22:55:30.569Z INFO module/wrapper.go:259 Error fetching data for metricset elasticsearch.enrich: HTTP error 403 in : 403 Forbidden
I'm aware of this topic that was raised a few years ago, but I've checked, all the elasticsearch modules in modules.d are disabled.
My understanding was commenting out the metricsets in the module config ensured they were removed from that beat's monitoring profile.
Everything is running on CentOS 7 using the rpm packages. I use the Elastic ansible galaxy roles to configure Elasticsearch and Metricbeat. The monitored cluster is running on 7.8.0 (long story) and the remote monitoring cluster is running on 7.10.2.
Truly I am confused and could use some guidance.
Thank you,
Devin