# Getting lots of context canceled - internal error in APM server logs

**URL:** <https://discuss.elastic.co/t/getting-lots-of-context-canceled-internal-error-in-apm-server-logs/241582>\
**Category:** APM\
**Tags:** java, server\
**Created:** [July 17, 2020, 5:52am UTC](https://discuss.elastic.co/t/getting-lots-of-context-canceled-internal-error-in-apm-server-logs/241582 "2020-07-17T05:52:45Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tarund](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@tarund](https://discuss.elastic.co/u/tarund)\
**Post date:** [July 17, 2020, 5:52am UTC](https://discuss.elastic.co/t/getting-lots-of-context-canceled-internal-error-in-apm-server-logs/241582/1 "2020-07-17T05:52:45Z")

</div>

I am having the below setup  
**Elasticsearch version** : 7.5.1  
**APM Server version** : 7.5.1  
**APM Agent language and version** : elasticapm-java/1.12.0

I am seeing lot of the below errors -

```auto
2020-07-17T11:10:00.132+0530	ERROR	[request]	middleware/log_middleware.go:74	internal error	{"request_id": "cda7d079-df4e-4a1f-a2c8-f98d020bbb0b", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "2.12.27.3", "user-agent": "elasticapm-java/1.12.0", "response_code": 500, "error": "context canceled"}

```

It's about 10% of the events accepted/ingested.  
I tried debug logging as well, but could not relate the error message to any specific event.  
Can u help me understand what does this error signify ? when does amp-server throw such error ?

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [July 17, 2020, 8:27am UTC](https://discuss.elastic.co/t/getting-lots-of-context-canceled-internal-error-in-apm-server-logs/241582/2 "2020-07-17T08:27:16Z")

</div>

Are there any other errors, or only "context canceled"? Any errors in the agent log? This server log message might mean that the server is timing out while reading requests from or writing responses to the agent.

---

<div class="post-metadata">

**Author:** ![tarund](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@tarund](https://discuss.elastic.co/u/tarund)\
**Post date:** [July 21, 2020, 5:46am UTC](https://discuss.elastic.co/t/getting-lots-of-context-canceled-internal-error-in-apm-server-logs/241582/3 "2020-07-21T05:46:19Z")

</div>

No errors in the agent log(u mean the application log on which the agent is running). Is there a way to log agent errors separately ?  
will this error imply data loss ?

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [July 21, 2020, 6:21am UTC](https://discuss.elastic.co/t/getting-lots-of-context-canceled-internal-error-in-apm-server-logs/241582/4 "2020-07-21T06:21:10Z")

</div>

> No errors in the agent log(u mean the application log on which the agent is running). Is there a way to log agent errors separately ?

You can configure the Java agent to log to a separate file: [Logging configuration options | APM Java Agent Reference [1.x] | Elastic](https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html#config-log-file)

> will this error imply data loss ?

Probably. You may need to [tune the APM Server performance](https://www.elastic.co/guide/en/apm/server/7.9/tune-apm-server.html) and/or [reduce the sampling rate](https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-transaction-sample-rate) to reduce load on the server.

---

<div class="post-metadata">

**Author:** ![tarund](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@tarund](https://discuss.elastic.co/u/tarund)\
**Post date:** [July 22, 2020, 4:23am UTC](https://discuss.elastic.co/t/getting-lots-of-context-canceled-internal-error-in-apm-server-logs/241582/5 "2020-07-22T04:23:22Z")

</div>

I found the below in agent logs --  
2020-07-22 00:04:07.019 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Read timed out, response code is -1  
2020-07-22 00:04:07.018 [TaskManagerPoolThread-82] TRACE co.elastic.apm.agent.impl.transaction.AbstractSpan - incrementing references at  
2020-07-22 00:04:07.019 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed

2020-07-22 00:04:32.757 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed  
java.net.SocketTimeoutException: Read timed out  
at java.net.SocketInputStream.socketRead0(Native Method)  
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)  
at java.net.SocketInputStream.read(SocketInputStream.java:171)  
at java.net.SocketInputStream.read(SocketInputStream.java:141)  
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)  
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)  
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)  
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)  
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)  
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)  
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)  
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.flush(IntakeV2ReportingEventHandler.java:224)  
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:128)  
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)  
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)  
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)  
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)  
at java.lang.Thread.run(Thread.java:748)  
2020-07-22 00:04:32.758 [apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null  
2020-07-22 00:04:32.758 [apm-reporter] ERROR co.elastic.apm.agent.report.HttpUtils - Exception when closing input stream of HttpURLConnection.

2020-07-22 00:05:32.919 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Error writing request body to server, response code is -1

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 12, 2020, 12:23am UTC](https://discuss.elastic.co/t/getting-lots-of-context-canceled-internal-error-in-apm-server-logs/241582/6 "2020-08-12T00:23:27Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
