# Ingesting Prometheus data via remote\_write with Elastic Agent (EKS) **URL:** \ **Category:** Elastic Agent\ **Created:** [May 8, 2025, 10:36pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969 "2025-05-08T22:36:00Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![trudesea](https://avatars.discourse-cdn.com/v4/letter/t/e480ec/32.png) [@trudesea](https://discuss.elastic.co/u/trudesea)\ **Post date:** [May 8, 2025, 10:36pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969/1 "2025-05-08T22:36:00Z")
So I have a EKS cluster using the Prometheus helm chart. I've followed this doc: > **[Prometheus Integration | Elastic Documentation](https://www.elastic.co/docs/reference/integrations/prometheus#prometheus-server-remote-write)** > > This integration can collect metrics from: Prometheus Exporters (Collectors), Prometheus Server Remote-Write, Prometheus Queries (PromQL). The Prometheus... But this documentation is well, atrocious. I assume the Host in the provided screenshot is the Prometheus server that is configured to remote write. ![Screenshot 2025-05-08 182352](https://us1.discourse-cdn.com/elastic/original/3X/e/c/ecefd06aad8afe1012faf52b89255b54e3abc9d6.png) On the Prometheus side, I have this configuration: ![Screenshot 2025-05-08 183119](https://us1.discourse-cdn.com/elastic/original/3X/c/0/c0a72670246a078831a42e748df43f4b24d47fe1.png) On the Prometheus integration configuration page, the data should be going to the datastream: prometheus.remote\_write I do not see that particular datastream. Is there a way to make sure the elastic agent is getting data from the Prometheus server? Thanks ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 8, 2025, 11:59pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969/2 "2025-05-08T23:59:09Z")
> [@trudesea](#): > > On the Prometheus integration configuration page, the data should be going to the datastream: prometheus.remote\_write Just a correction, the data stream would be something like `metrics-prometheus.remote_write-default` , do you have anything like this? If not, can prometheus reach the elastic agent port? Also, if I'm not wrong on the _Host_ configuration of the integration you should use the IP address it will listen, normally you just use `0.0.0.0` to listen on all IP availables. ---
**Author:** ![trudesea](https://avatars.discourse-cdn.com/v4/letter/t/e480ec/32.png) [@trudesea](https://discuss.elastic.co/u/trudesea)\ **Post date:** [May 9, 2025, 2:54pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969/3 "2025-05-09T14:54:58Z")
Ok, so I changed the host to 0.0.0.0 and I have a new datastream: metrics-prometheus.remote\_write-default I am seeing a lot of indexing errors on the agents so I need to investigate that. So this is what I'm seeing on the prometheus server: Failed to send batch, retrying" component=remote remote\_name=1f86d2 url=[http://elastic-agent.kube-system:9201/write](http://elastic-agent.kube-system:9201/write) err="Post "[http://elastic-agent.kube-system:9201/write\](http://elastic-agent.kube-system:9201/write%5C)": dial tcp 10.100.189.151:9201: connect: connection refused" Not sure where to look next on that one. My guess is something wrong with the service manifest or something. (I do not have TLS enabled) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 10, 2025, 4:32pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969/4 "2025-05-10T16:32:21Z")
I would exec into the elastic agent container and check the status of the agent `elastic-agent status` Let us know what you see Also from the Prometheus server, can you curl the agent endpoint? ---
**Author:** ![trudesea](https://avatars.discourse-cdn.com/v4/letter/t/e480ec/32.png) [@trudesea](https://discuss.elastic.co/u/trudesea)\ **Post date:** [May 13, 2025, 1:46pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969/5 "2025-05-13T13:46:10Z")
Thanks for your help, I figured it out, with both the kubernetes and prometheus integrations, the agent pods where hitting their memory limits and in went into a cycle of OOMKilled and restarts. Had to limit them to 1.5G and they are using most of that. Now the only issue I have is dropped data on the prometheus side but this is not an elastic issue. `msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=serviceMonitor/default/gitlab-webservice/0 target=http://172.32.1.6:8083/metrics num_dropped=1` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 13, 2025, 6:40pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969/6 "2025-05-13T18:40:10Z")
Good ole OOM .. UGH... still to hard to find that.. what version are you on .. I thought we started to do better OOM detection? ---
**Author:** ![trudesea](https://avatars.discourse-cdn.com/v4/letter/t/e480ec/32.png) [@trudesea](https://discuss.elastic.co/u/trudesea)\ **Post date:** [May 14, 2025, 4:18pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969/7 "2025-05-14T16:18:44Z")
We are still on 8.14.3. Upgrades coming soon. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 14, 2025, 4:42pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969/8 "2025-05-14T16:42:21Z")
I will say OOM is easy to see with K9s for any pods ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 14, 2025, 5:28pm UTC](https://discuss.elastic.co/t/ingesting-prometheus-data-via-remote-write-with-elastic-agent-eks/377969/9 "2025-05-14T17:28:23Z")
If I'm not wrong, version 8.14.X has some problems related to memory leaks on metricbeat. We skipped it for this reason.