Hello,
We are currently using Elastic Agent with the integration Elasticsearch, everything works as expected except the Shard metrics which seems to be written randomly into the cluster.
We are using ES 8.17.3 with Elasticsearch integration 1.15.3. (This issue already existed previously with ES 8.15.5).
The shards metrics is set to send every 5m (changed for testing purposes, but it was set to 1m), Elasticsearch integration on all nodes of the cluster are set as node (if set in cluster, it only duplicates some other metrics, this one is however still the same).
I've tried to find the culprit, but there is no error in the Elastic agent logs, nor in the Elasticsearch logs. Moreover the Metricbeat component is able to send the data to Elasticsearch, I've determined it with the Elastic agent metricbeat component that I set to send these specific shard metrics every 5min, while the others are every minute, and every 5 minutes metricbeat send 3 bulk requests instead of 1 per node: (See first printscreen of image below)
I've also executed the diagnostic and the shards are visible.
A manual query to the endpoint for shards only takes around 1s.
I've checked multiple logs, including ingest pipelines which don't have any errors, maybe it is unable to index but one of the ingest pipeline is just ignoring it?
While the data is received randomly (See second print screen in the image linked)
Is it a behavior by design to randomly send this data, is it a misconfiguration or is there a bug?
I've been unable to find any information about this, and the documentation doesn't mention any of this. Also this is probably a miss, but even the documentation indicating which API is queried is redirecting to an old Elasticsearch version 6.2: integrations/packages/elasticsearch at main · elastic/integrations · GitHub.
Any help would be appreciated, thanks.
