APM Java Agent error

Elastic Observability APM
1

Hi,
short question:
can someone tell my what is wrong with my app or with my apm server?

I get this error:

[elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 400 for URL: https://61f12ad7e3554c5caa26da35e9044fda.elastic-nonprod.services.nublar.de:9243/intake/v2/events , response code is 400 2021-04-28 14:07:30,883 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - { "accepted": 79, "errors": [ { "message": "validation error: metricset: 'samples' required", "document": "{\"metricset\":{\"timestamp\":1619618850159000,\"tags\":{\"application\":\"mpi\",\"path\":\"/app/product/mpi/4.0.3.3/apache-tomcat-9.0.37\"},\"samples\":{}}}" } ] }

thanks alot

1 Like
2

Hi @apox
what version of apm-server, elastic-apm-agent do you use?

3

Hi @apox, there are a few known cases where the apm server returns a 400 status in documentation, did you tried to check for those already ?

4

Hello,
@nugusbayevkk we are using APM Server 7.12.0 and Agent 1.X

@Sylvain_Juge yes with checkt this page but nothings helps. what we dont understand is this error

"errors": [ { "message": "validation error: metricset: 'samples' required",

from were comes metricset samples required ?

5

There are metrics that are automatically captured by the agent, in seems in your case that some of them are sent with an incompatible format to the APM server ?

Can you provide the full version of the Elastic APM Java agent used here ? There are quite a few in the 1.x range. The simplest would be to update to the latest 1.23.0 as of today just to be sure it's not something that has been already fixed in the agent.

If updating the agent does not make the problem go away, then the next step is to set log_level=debug and to see which metrics are being sent for more details.

6

Hi all! We have been facing the same issue with different Java Agent versions. For us it is connected to Kafka, but reading this thread I'm assuming that it is a more general problem?
APM Java agent v1.20.0, v1.22.0 and v1.23.0 with APM Server version 7.11.1
The error messages on our side are connected to Kafka samples which (sometimes) seem to be missing. The error messages didn't change after updating the agent version.

	co.elastic.apm.agent.report.IntakeV2ReportingEventHandler: Error sending data to APM server: Server returned HTTP response code: 400 for URL: https://apm.our-domain.example.com:9243/intake/v2/events, response code is 400

    {
      "accepted": 145,
      "errors": [
        {
          "message": "validation error: metricset: 'samples' required",
          "document": "{\"metricset\":{\"timestamp\":1620368999667000,\"tags\":{\"kafka_version\":\"2.6.1\",\"record_cache_id\":\"products\",\"spring_id\":\"defaultKafkaStreamsBuilder\",\"task_id\":\"0_3\",\"thread_id\":\"stream-4e90d46e-446b-4cf1-a1e6-241ee608f5d7-StreamThread-1\"},\"samples\":{}}}"
        },
        {
          "message": "validation error: metricset: 'samples' required",
          "document": "{\"metricset\":{\"timestamp\":1620368999667000,\"tags\":{\"kafka_version\":\"2.6.1\",\"record_cache_id\":\"products\",\"spring_id\":\"defaultKafkaStreamsBuilder\",\"task_id\":\"0_7\",\"thread_id\":\"stream-4e90d46e-446b-4cf1-a1e6-241ee608f5d7-StreamThread-1\"},\"samples\":{}}}"
        }
      ]
    }

Cheers,
Jan

7

Hi !

The issue with empty metrics sample has been fixed with Avoid serializing metric-sets with no samples by eyalkoren · Pull Request #1826 · elastic/apm-agent-java · GitHub.
It's now merged on master and will be included in the next release (likely due in a few days at worst), make sure to subscribe to GH notifications for that.

In the mean time, you can use the latest snapshot that already includes this fix.

1 Like
8

Thank you! Updating the Java agent to 1.24.0 did the trick on our side :slight_smile: (still with server version 7.11.1)

1 Like
9

Thanks man now its working for us

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.