Hello,
We are scraping metrics from some 30+ Kafka clusters running in Confluent Cloud.
We are scraping the metrics api using prometheus.
Prometheus is configured to remote_write to an elastic agent running among others, the prometheus integration with remote_write enabled.
Metrics are comming in, and we can visualize them in Kibana. But there is a consistent delay of around 5 mins.
In both Prometheus and Grafana there are no delays, so i am thinking it has to do with my remote write config in prometheus.
The queue config is mortly set to the default settings as described in the prometheus docs.
Is there something I can tweak on either the agent or in prometheus to eliminate this delay?
Best regards
Christian Oelsner