Metricbeat events not indexed in Elasticsearch

[Hugues_Bernard]

I configured metricbeat on one node of a 3 nodes elasticsearch cluster. Metricbeat output is set to a separate elasticsearch on a dedicated VM used for monitoring data only. Metricbeat is sending events every 10 seconds without any error, as seen in its log set in debug level (log says PublishEvents: 1 events have been published to elasticsearch in 248.464504ms.). However, on the monitoring elasticsearch node, no data is ever inserted in the metricbeat index. docs.count stays 0, as reported by _cat/indices:

curl "$(hostname):9200/_cat/indices?v&pretty"
health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   metricbeat-7.9.1-2021.06.04-000001 nKxgLXjlQXyRf0bkdTUWnQ   1   1          0            0       208b           208b
green  open   .monitoring-es-7-mb-2021.06.04     H8-4y7VESXKtWrgcn0aRyQ   1   0          7            0    121.1kb        121.1kb
green  open   .apm-custom-link                   arYGemzhQbG0xK70_VfiHQ   1   0          0            0       208b           208b
green  open   .kibana_task_manager_1             P6t1VznFTbucXiwYpo14sg   1   0          6           75     45.5kb         45.5kb
green  open   .kibana-event-log-7.9.1-000001     bx5yWdw4SoyXnxDshPftSQ   1   0          1            0      5.5kb          5.5kb
green  open   .apm-agent-configuration           YWGN5b3UTC-sSwKtfwJixQ   1   0          0            0       208b           208b
green  open   .kibana_1                          TDF9qiolTAuF8O_-PsglDg   1   0         15            0     10.4mb         10.4mb

Kibana is also installed and running fine on the monitoring node, and also shows no data in the metricbeat index.

I see no errors reported anywhere. metricbeart.yml is damn simple:

output.elasticsearch:
  # monitoring elasticsearch node
  hosts: ["http://<obfuscated>:9200"] 

processors:
  - add_host_metadata: ~

logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat

The problem is probably on the elasticsearch monitoring node side but I am out of ideas. The elasticsearch installation is also very simple, and I made sure the exact same versions of all components are used.

Have you any idea/suggestion what could be wrong ?
Thanks already for your help.
Hugues

[angelo]

In newer versions of Elasticsearch, a refresh is not done unless the index has received a search request ... and subsequently, the stats may not be getting updated to show the current doc count in the index. It sounds like this may be the case here.

You should be able to confirm by either calling the _search API or the _refresh API on that index and then retry the _cat/indices call to see the updated doc counts. The _search API should also report the updated total hits as well in the response assuming it returns the default 10 docs.

[Hugues_Bernard]

Thanks Angelo for your reply.
I was fooled by the "metricbeat*" index: it is not the one that receives monitoring events. The one to look at is ".monitoring_es-*".
The root cause why no monitoring data was visible in Kibana is the fact that metricbeat was not running on the master node of the monitored cluster, but only on non-master nodes. Until metricbeat is set-up and running on a master node, Kibana dashboard seems not able to display cluster monitoring data.

For info: I found out that reading comments on kibana show elasticsearch no monitoring data , but exists index in elasticsearch · Issue #22483 · elastic/kibana · GitHub :

The cluster_stats document is produced by the elected master node of the monitored cluster, and if it's not present the UI does not recognize other data because it needs the cluster_stats to represent the cluster. We're looking at ways to address this issue separately.

Thanks for your help