# Sending payload to APM server failed **URL:** \ **Category:** APM\ **Tags:** java, server\ **Created:** [August 5, 2022, 5:32pm UTC](https://discuss.elastic.co/t/sending-payload-to-apm-server-failed/311562 "2022-08-05T17:32:36Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![tarund](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@tarund](https://discuss.elastic.co/u/tarund)\ **Post date:** [August 5, 2022, 5:32pm UTC](https://discuss.elastic.co/t/sending-payload-to-apm-server-failed/311562/1 "2022-08-05T17:32:36Z")
**Elasticsearch version** : 7.15.2 **APM Server version** : 7.15.0 **APM Agent language and version** : Java Elastic APM Agent 1.10 **Original install method (e.g. download page, yum, deb, from source, etc.) and version**:rpm **Fresh install or upgraded from other version?** fresh install **Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: I am seeing the below error in the APM Agent log at regular intervals. But I am still able to see metric and transaction events in the apm server. I don't see any error in the APM server log, no queue full error either. So, the server is not under heavy load. How can I know if events are lost? ```auto 2022-08-05 22:31:47.079 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Flushing 5826 uncompressed 1481 compressed bytes 2022-08-05 22:31:47.081 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%) 2022-08-05 22:31:47.082 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Unexpected end of file from server, response code is -1 2022-08-05 22:31:47.082 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed java.net.SocketException: Unexpected end of file from server at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:855) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1595) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500) at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.flush(IntakeV2ReportingEventHandler.java:224) at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:128) at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116) at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50) at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) 2022-08-05 22:31:47.083 [apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null 2022-08-05 22:31:47.083 [apm-reporter] ERROR co.elastic.apm.agent.report.HttpUtils - Exception when closing input stream of HttpURLConnection. ``` Below is the elasticapm.properties file ```auto instrument=true transaction_sample_rate=1.0 transaction_max_spans=1 stack_trace_limit=5 trace_methods=com.net.kafka.processor.*#* application_packages=com.net server_urls=http://192.168.1.1:8200 capture_headers=false span_frames_min_duration=700ms metrics_interval=20s max_queue_size=2000 log_file=/var/log/elastic-apm.log log_level=TRACE log_file_size=100mb server_timeout=30s api_request_size=3mb api_request_time=60s ``` ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [August 8, 2022, 7:09am UTC](https://discuss.elastic.co/t/sending-payload-to-apm-server-failed/311562/2 "2022-08-08T07:09:56Z")
Two things to try out before diagnosing further: - Lower `api_request_time` on the agent side or higher [`read_timeout`](https://www.elastic.co/guide/en/apm/server/7.15/configuration-process.html#read_timeout) on the APM Server side - Lower `api_request_size` on the agent side or higher [`max_event_size`](https://www.elastic.co/guide/en/apm/server/7.15/configuration-process.html#max_event_size) on the APM Server side (though they seem to match) - Upgrade your agent to a later version, preferably latest ---
**Author:** ![tarund](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@tarund](https://discuss.elastic.co/u/tarund)\ **Post date:** [August 8, 2022, 7:53am UTC](https://discuss.elastic.co/t/sending-payload-to-apm-server-failed/311562/3 "2022-08-08T07:53:54Z")
@Eyal_Koren , thanks for your revert. `api_request_time` & `api_request_size` were set to default initially, still giving errors, we had increased it to above values to see if that helps. `read_timeout` and `max_event_size` are set to 90s and 1million respectively. We have tried with higher version of agent, but we got errors in instrumentation of certain classes since the application is build with lower jdk version ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [August 8, 2022, 8:27am UTC](https://discuss.elastic.co/t/sending-payload-to-apm-server-failed/311562/4 "2022-08-08T08:27:24Z")
> [@tarund](#): > > `read_timeout` and `max_event_size` are set to 90s and 1million respectively Is there anything between the agent and the APM Server (proxy, load-balancer, firewall etc.) that may be timing out these requests? Please try with the default of both APM Server and agent and see if the issue gets reproduced, even using some test APM Server > We have tried with higher version of agent, but we got errors in instrumentation of certain classes since the application is build with lower jdk version You shouldn't get errors that you don't get with 1.10.0 as far as I can tell. Could those be `debug` messages about version mismatch that can be safely ignored? If you can share these errors, we may be able to tell what that is. ---
**Author:** ![tarund](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@tarund](https://discuss.elastic.co/u/tarund)\ **Post date:** [August 9, 2022, 6:54am UTC](https://discuss.elastic.co/t/sending-payload-to-apm-server-failed/311562/5 "2022-08-09T06:54:51Z")
> [@Eyal\_Koren](#): > > You shouldn't get errors that you don't get with 1.10.0 as far as I can tell. Could those be `debug` messages about version mismatch that can be safely ignored? If you can share these errors, we may be able to tell what that is. Please see the below error when we tried agent 1.18.0 ```auto 2022-07-21 17:53:29,813 [main] ERROR co.elastic.apm.agent.util.DependencyInjectingServiceLoader - unable to instantiate 'co.elastic.apm.agent.httpclient.HttpRequestHeadersInstrumentation', unsupported class version error: co/elastic/apm/agent/httpclient/HttpRequestHeadersInstrumentation has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 ``` ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [August 9, 2022, 8:40am UTC](https://discuss.elastic.co/t/sending-payload-to-apm-server-failed/311562/6 "2022-08-09T08:40:38Z")
Yes, this was changed to `debug` almost [two years ago](https://github.com/elastic/apm-agent-java/pull/1397) in [1.18.1](https://www.elastic.co/guide/en/apm/agent/java/current/release-notes-1.x.html#_bug_fixes_23). Why are you trying such an old agent and not the latest? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 30, 2022, 4:40am UTC](https://discuss.elastic.co/t/sending-payload-to-apm-server-failed/311562/7 "2022-08-30T04:40:48Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.