I configured metricbeat on one node of a 3 nodes elasticsearch cluster. Metricbeat output is set to a separate elasticsearch on a dedicated VM used for monitoring data only. Metricbeat is sending events every 10 seconds without any error, as seen in its log set in debug level (log says
PublishEvents: 1 events have been published to elasticsearch in 248.464504ms.). However, on the monitoring elasticsearch node, no data is ever inserted in the metricbeat index.
0, as reported by
curl "$(hostname):9200/_cat/indices?v&pretty" health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open metricbeat-7.9.1-2021.06.04-000001 nKxgLXjlQXyRf0bkdTUWnQ 1 1 0 0 208b 208b green open .monitoring-es-7-mb-2021.06.04 H8-4y7VESXKtWrgcn0aRyQ 1 0 7 0 121.1kb 121.1kb green open .apm-custom-link arYGemzhQbG0xK70_VfiHQ 1 0 0 0 208b 208b green open .kibana_task_manager_1 P6t1VznFTbucXiwYpo14sg 1 0 6 75 45.5kb 45.5kb green open .kibana-event-log-7.9.1-000001 bx5yWdw4SoyXnxDshPftSQ 1 0 1 0 5.5kb 5.5kb green open .apm-agent-configuration YWGN5b3UTC-sSwKtfwJixQ 1 0 0 0 208b 208b green open .kibana_1 TDF9qiolTAuF8O_-PsglDg 1 0 15 0 10.4mb 10.4mb
Kibana is also installed and running fine on the monitoring node, and also shows no data in the metricbeat index.
I see no errors reported anywhere.
metricbeart.yml is damn simple:
output.elasticsearch: # monitoring elasticsearch node hosts: ["http://<obfuscated>:9200"] processors: - add_host_metadata: ~ logging.level: debug logging.to_files: true logging.files: path: /var/log/metricbeat name: metricbeat
The problem is probably on the elasticsearch monitoring node side but I am out of ideas. The elasticsearch installation is also very simple, and I made sure the exact same versions of all components are used.
Have you any idea/suggestion what could be wrong ?
Thanks already for your help.