Ingesting Prometheus data via remote_write with Elastic Agent (EKS)

Elastic Stack Elastic Agent
1

So I have a EKS cluster using the Prometheus helm chart. I've followed this doc:

But this documentation is well, atrocious. I assume the Host in the provided screenshot is the Prometheus server that is configured to remote write.

Screenshot 2025-05-08 182352
Screenshot 2025-05-08 1823521320×671 31.7 KB

On the Prometheus side, I have this configuration:

Screenshot 2025-05-08 183119
Screenshot 2025-05-08 183119384×415 68.4 KB

On the Prometheus integration configuration page, the data should be going to the datastream: prometheus.remote_write

I do not see that particular datastream. Is there a way to make sure the elastic agent is getting data from the Prometheus server?

Thanks

2

Just a correction, the data stream would be something like metrics-prometheus.remote_write-default , do you have anything like this?

If not, can prometheus reach the elastic agent port? Also, if I'm not wrong on the Host configuration of the integration you should use the IP address it will listen, normally you just use 0.0.0.0 to listen on all IP availables.

3

Ok, so I changed the host to 0.0.0.0 and I have a new datastream: metrics-prometheus.remote_write-default

I am seeing a lot of indexing errors on the agents so I need to investigate that.

So this is what I'm seeing on the prometheus server:
Failed to send batch, retrying" component=remote remote_name=1f86d2 url=http://elastic-agent.kube-system:9201/write err="Post "http://elastic-agent.kube-system:9201/write\": dial tcp 10.100.189.151:9201: connect: connection refused"

Not sure where to look next on that one. My guess is something wrong with the service manifest or something. (I do not have TLS enabled)

4

I would exec into the elastic agent container and check the status of the agent

elastic-agent status

Let us know what you see

Also from the Prometheus server, can you curl the agent endpoint?