Sending payload to APM server failed

Elastic Observability APM
,
1

Elasticsearch version: 7.15.2

APM Server version: 7.15.0

APM Agent language and version: Java Elastic APM Agent 1.10

Original install method (e.g. download page, yum, deb, from source, etc.) and version:rpm

Fresh install or upgraded from other version? fresh install

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I am seeing the below error in the APM Agent log at regular intervals. But I am still able to see metric and transaction events in the apm server. I don't see any error in the APM server log, no queue full error either. So, the server is not under heavy load. How can I know if events are lost?


2022-08-05 22:31:47.079 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Flushing 5826 uncompressed 1481 compressed bytes
2022-08-05 22:31:47.081 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
2022-08-05 22:31:47.082 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Unexpected end of file from server, response code is -1
2022-08-05 22:31:47.082 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed
java.net.SocketException: Unexpected end of file from server
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:855)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1595)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
	at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.flush(IntakeV2ReportingEventHandler.java:224)
	at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:128)
	at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
	at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
	at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
	at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
	at java.lang.Thread.run(Thread.java:750)
2022-08-05 22:31:47.083 [apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
2022-08-05 22:31:47.083 [apm-reporter] ERROR co.elastic.apm.agent.report.HttpUtils - Exception when closing input stream of HttpURLConnection.

Below is the elasticapm.properties file


instrument=true
transaction_sample_rate=1.0
transaction_max_spans=1
stack_trace_limit=5
trace_methods=com.net.kafka.processor.*#*
application_packages=com.net
server_urls=http://192.168.1.1:8200
capture_headers=false
span_frames_min_duration=700ms
metrics_interval=20s
max_queue_size=2000
log_file=/var/log/elastic-apm.log
log_level=TRACE
log_file_size=100mb
server_timeout=30s
api_request_size=3mb
api_request_time=60s
2

Two things to try out before diagnosing further:

  • Lower api_request_time on the agent side or higher read_timeout on the APM Server side
  • Lower api_request_size on the agent side or higher max_event_size on the APM Server side (though they seem to match)
  • Upgrade your agent to a later version, preferably latest
3

@Eyal_Koren , thanks for your revert.

api_request_time & api_request_size were set to default initially, still giving errors, we had increased it to above values to see if that helps.

read_timeout and max_event_size are set to 90s and 1million respectively.

We have tried with higher version of agent, but we got errors in instrumentation of certain classes since the application is build with lower jdk version

4

Is there anything between the agent and the APM Server (proxy, load-balancer, firewall etc.) that may be timing out these requests? Please try with the default of both APM Server and agent and see if the issue gets reproduced, even using some test APM Server

We have tried with higher version of agent, but we got errors in instrumentation of certain classes since the application is build with lower jdk version

You shouldn't get errors that you don't get with 1.10.0 as far as I can tell. Could those be debug messages about version mismatch that can be safely ignored? If you can share these errors, we may be able to tell what that is.

5

Please see the below error when we tried agent 1.18.0


2022-07-21 17:53:29,813 [main] ERROR co.elastic.apm.agent.util.DependencyInjectingServiceLoader - unable to instantiate 'co.elastic.apm.agent.httpclient.HttpRequestHeadersInstrumentation', unsupported class version error: co/elastic/apm/agent/httpclient/HttpRequestHeadersInstrumentation has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
6

Yes, this was changed to debug almost two years ago in 1.18.1.
Why are you trying such an old agent and not the latest?

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.