400 response on /intake/v2/events after elastic-agent 8.11.x upgrade

[viktorbard (Viktor)]

Hello Elastic team!

I'm having issues with apm java agent after upgrading elasic-agent acting as APM server to 8.11.x. See specs below.

Kibana version: 8.11.1

Elasticsearch version: 8.11.1

APM Server version: 8.11.1

APM Agent language and version: java 1.43.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: elastic agent via docker, java agent downloaded from co.elastic.apm maven source.

Fresh install or upgraded from other version?: Upgraded elastic-agent from 8.10.4

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

After upgrading elastic agent running APM Server from 8.10.4 -> 8.11.x all services running the APM java agent starts throwing error provided below

Steps to reproduce:

1. Install APM Server with elastic agent docker image v8.10.4.
2. Run APM java agent v1.43.0 along with a java service.
3. Upgrade elastic-agent from 8.10.4 -> 8.11.x

Provide logs and/or server output (if relevant):
</> [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Server returned HTTP response code: 400 for URL: http://[elastic-agent-url]:8200/intake/v2/events, response code is 400 </>

Please let me know if you need additional information.

[Sylvain_Juge]

Hi, welcome to our forum @viktorbard !

As you seem to have updated the apm-server, I would suggest to start by looking at the APM server troubleshooting documentation: Troubleshoot | APM User Guide [8.11] | Elastic

In particular, there is a whole section about common 4xx errors here: APM Server response codes | APM User Guide [8.11] | Elastic

As all your services are throwing the same exception, I think it's more related to the APM server than the APM agents that are deployed on your application.

[viktorbard (Viktor)]

I could not find out the reason for this issue but after upgrading java APM agents to 1.44.0 the 400 error doesn't appear so it seems to me like some compatibility issues.