Elastic APM - Error sending data to APM server: Read timed out, response code is -1

Elastic Observability APM
,
1

I have searched for the error I shared below in many places, but I haven't come across a complete solution. Since other threads are locked, I couldn't leave a comment. Could you help me with this issue?

Kibana version: 8.15.0

Elasticsearch version: 8.15.0

APM Server version: 8.15.0

APM Agent language and version: Java 1.52.2

Browser version: Google Chrome 133.0.6943.141

Original install method (e.g. download page, yum, deb, from source, etc.) and version: yum

Fresh install or upgraded from other version? I have just upgraded the Elastic APM Agent JAR from 1.49.0 to 1.52.2. However, this method did not resolve the issue.

Is there anything special in your setup? Yes, there are 2 APM Fleet machines, and there is Nginx in front of them. It goes from Nginx to HAProxy. I have previously changed the agent configuration to fix the issue.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:
Elastic APM integration is being done with APM Fleet. After the application starts, it throws the following errors immediately or within an hour.

Provide logs and/or server output (if relevant):

2025-02-27 16:59:45,483 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: null
2025-02-27 16:59:45,483 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 36 seconds (+/-10%)
2025-02-27 17:00:19,215 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1
2025-02-27 17:00:19,215 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Sending payload to APM server failed
java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:115) ~[?:?]
        at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[?:?]
        at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1132) ~[?:?]
        at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:828) ~[?:?]
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[?:?]
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:292) ~[?:?]
        at java.io.BufferedInputStream.read(BufferedInputStream.java:351) ~[?:?]
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:746) ~[?:?]
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1604) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509) ~[?:?]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:246) ~[?:?]
        at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:207) [elastic-apm-agent-1.52.2.jar:1.52.2-SNAPSHOT]
        at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:180) [elastic-apm-agent-1.52.2.jar:1.52.2-SNAPSHOT]
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:90) [elastic-apm-agent-1.52.2.jar:1.52.2-SNAPSHOT]
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:39) [elastic-apm-agent-1.52.2.jar:1.52.2-SNAPSHOT]
        at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [elastic-apm-agent-1.52.2.jar:1.52.2-SNAPSHOT]
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [elastic-apm-agent-1.52.2.jar:1.52.2-SNAPSHOT]
        at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent-1.52.2.jar:1.52.2-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:835) [?:?]
2

Hello @rbkuscu. My understanding of the error you're seeing java.net.SocketTimeoutException: Read timed out is that it takes too long for this APM Java Agents to get a response from APM Server. This suggests APM Server or the network connection between the agents and APM Server has issues that prevent a successful communication.

You mention you upgraded from 1.49.0 to 1.52.2. Was this error present also before?

If I understood correctly your setup you have HAProxy facing users making requests to this service. Then Nginx and then 2 services, each instrumented separately with APM Java Agent.
You then mention that you're using Elastic APM integration on Fleet, but is not clear if this is an on-prem deployment or are you using Elastic Cloud Hosted.

What I would suggest is:

  • check if connectivity works as expected between your services and the APM Server endpoint they need to send data to.
  • check that APM Server your sending data to is healthy and able to respond to requests.

Is possible that APM Server is overloaded or gets overloaded as data is sent to it. If this happens it can slow down responding to clients and trigger the behavior you're seeing. Usually APM Server gets overloaded due to backpressure from Elasticsearch when writing data to it because the Elasticsearch cluster is overloaded as well, so I would also suggest to look into the status of the cluster and APM related data streams.