Ingesting Prometheus data via remote_write with Elastic Agent (EKS)

Elastic Stack Elastic Agent
1

So I have a EKS cluster using the Prometheus helm chart. I've followed this doc:

But this documentation is well, atrocious. I assume the Host in the provided screenshot is the Prometheus server that is configured to remote write.

Screenshot 2025-05-08 182352
Screenshot 2025-05-08 1823521320×671 31.7 KB

On the Prometheus side, I have this configuration:

Screenshot 2025-05-08 183119
Screenshot 2025-05-08 183119384×415 68.4 KB

On the Prometheus integration configuration page, the data should be going to the datastream: prometheus.remote_write

I do not see that particular datastream. Is there a way to make sure the elastic agent is getting data from the Prometheus server?

Thanks

2

Just a correction, the data stream would be something like metrics-prometheus.remote_write-default , do you have anything like this?

If not, can prometheus reach the elastic agent port? Also, if I'm not wrong on the Host configuration of the integration you should use the IP address it will listen, normally you just use 0.0.0.0 to listen on all IP availables.

3

Ok, so I changed the host to 0.0.0.0 and I have a new datastream: metrics-prometheus.remote_write-default

I am seeing a lot of indexing errors on the agents so I need to investigate that.

So this is what I'm seeing on the prometheus server:
Failed to send batch, retrying" component=remote remote_name=1f86d2 url=http://elastic-agent.kube-system:9201/write err="Post "http://elastic-agent.kube-system:9201/write\": dial tcp 10.100.189.151:9201: connect: connection refused"

Not sure where to look next on that one. My guess is something wrong with the service manifest or something. (I do not have TLS enabled)

4

I would exec into the elastic agent container and check the status of the agent

elastic-agent status

Let us know what you see

Also from the Prometheus server, can you curl the agent endpoint?

5

Thanks for your help,

I figured it out, with both the kubernetes and prometheus integrations, the agent pods where hitting their memory limits and in went into a cycle of OOMKilled and restarts.

Had to limit them to 1.5G and they are using most of that.

Now the only issue I have is dropped data on the prometheus side but this is not an elastic issue.

msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=serviceMonitor/default/gitlab-webservice/0 target=http://172.32.1.6:8083/metrics num_dropped=1

1 Like
6

Good ole OOM .. UGH... still to hard to find that.. what version are you on .. I thought we started to do better OOM detection?

7

We are still on 8.14.3. Upgrades coming soon.

8

I will say OOM is easy to see with K9s for any pods

9

If I'm not wrong, version 8.14.X has some problems related to memory leaks on metricbeat.

We skipped it for this reason.

1 Like