Getting lots of context canceled - internal error in APM server logs

[tarund (Tarun)]

I am having the below setup
Elasticsearch version: 7.5.1
APM Server version: 7.5.1
APM Agent language and version: elasticapm-java/1.12.0

I am seeing lot of the below errors -

2020-07-17T11:10:00.132+0530	ERROR	[request]	middleware/log_middleware.go:74	internal error	{"request_id": "cda7d079-df4e-4a1f-a2c8-f98d020bbb0b", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "2.12.27.3", "user-agent": "elasticapm-java/1.12.0", "response_code": 500, "error": "context canceled"}

It's about 10% of the events accepted/ingested.
I tried debug logging as well, but could not relate the error message to any specific event.
Can u help me understand what does this error signify ? when does amp-server throw such error ?

[axw (Andrew Wilkins)]

Are there any other errors, or only "context canceled"? Any errors in the agent log? This server log message might mean that the server is timing out while reading requests from or writing responses to the agent.

[tarund (Tarun)]

No errors in the agent log(u mean the application log on which the agent is running). Is there a way to log agent errors separately ?
will this error imply data loss ?

[axw (Andrew Wilkins)]

No errors in the agent log(u mean the application log on which the agent is running). Is there a way to log agent errors separately ?

You can configure the Java agent to log to a separate file: Logging configuration options | APM Java Agent Reference [1.x] | Elastic

will this error imply data loss ?

Probably. You may need to tune the APM Server performance and/or reduce the sampling rate to reduce load on the server.

[tarund (Tarun)]

I found the below in agent logs --
2020-07-22 00:04:07.019 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Read timed out, response code is -1
2020-07-22 00:04:07.018 [TaskManagerPoolThread-82] TRACE co.elastic.apm.agent.impl.transaction.AbstractSpan - incrementing references at
2020-07-22 00:04:07.019 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed

2020-07-22 00:04:32.757 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.flush(IntakeV2ReportingEventHandler.java:224)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:128)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:748)
2020-07-22 00:04:32.758 [apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
2020-07-22 00:04:32.758 [apm-reporter] ERROR co.elastic.apm.agent.report.HttpUtils - Exception when closing input stream of HttpURLConnection.

2020-07-22 00:05:32.919 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Error writing request body to server, response code is -1