Elastic stack v8.13.3
apm agent：elastic-apm-agent-1.52.1.jar
I found the ERROR in the java-agent log.
http response code 503 and queue is full are two issues. I checked that queue.event_size should be set in the apm server configuration file, but I did not find the relevant configuration in the v8.13 documentation. What about http response code 503?
2025-03-09 08:41:01,610 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: {
"accepted": 0,
"errors": [
{
"message": "cannot handle stream: cannot process batch: queue is full"
}
]
}
2025-03-09 08:41:01,610 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%)
2025-03-09 08:41:01,610 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://xxxxx:8200/intake/v2/events, response code is 503
The partial configuration of java-agent is as follows:
server_urls=http://xxxxxxx:8200,http://xxxxxxx:8200,http://xxxxx:8200,http://xxxxxx:8200
transaction_max_spans=500
transaction_sample_rate=1
In kibana monitoring UI, it can be seen that the response error of APM server only causes the error caused by queue is full in the ingestion API, and no error caused by other reasons can be seen.
One more question to ask, I set transaction_sample_rate=0.3 via the remote proxy configuration, but I queried the Storage Explorer for nearly a day of data and saw that the sampling rate remained much higher than 30%
2025-03-07 15:11:05,671 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {transaction_sample_rate=0.3}
