kafka.consumer.bytes.consumed.total=NaN causes Error sending data to APM server: Server returned HTTP response code: 400

antonro · December 3, 2020, 4:02pm

Hi,
We started using Java APM agent and observe following errors in log files of our Java services with enabled agent. I see that some metrics are reported with NaN value which causes error with "expected number, but got string" description. Is there a way to resolve this or if not possible to suppress specific metric collection on agent side?
Thanks!

    2020-12-03 14:15:31,392 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)                                                                                                                                                
    2020-12-03 14:15:31,392 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 400 for URL: http://localhost:8200/intake/v2/events, response code is 400
    2020-12-03 14:15:31,392 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {                                                                                                                                                                                  
      "accepted": 56,                                                                                                                                                                                                                                                                                          
      "errors": [                                                                                                                                                                                                                                                                                              
        {                                                                                                                                                                                                                                                                                                      
          "message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/kafka.consumer.bytes.consumed.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string",
          "document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-1\",\"topic\":\"scheduler\"},\"samples\":{\"kafka.consumer.records.per.request.avg\":{\"value\":0.2890625},\"kafka.consumer.fetch.size.max\":{\"value\":360.0},\"kafka.consumer.bytes.consumed.total\ ":{\"value\":\"NaN\"},\"kafka.consumer.records.consumed.total\":{\"value\":\"NaN\"},\"kafka.consumer.fetch.size.avg\":{\"value\":97.5234375}}}}"
        },                                                                                                                                                                                                                                                                                                     
        {                                                                                                                                                                                                                                                                                                      
          "message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/kafka.consumer.fetch.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string",
          "document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-1\"},\"samples\":{\"kafka.consumer.commit.latency.avg\":{\"value\":0.0006666666666666666},\"kafka.consumer.heartbeat.rate\":{\"value\":0.3436819796082025},\"kafka.consumer.io-time-avg\":{\"value\": 0.00002053489579831933},\"kafka.consumer.last-heartbeat\":{\"value\":1.0},\"kafka.consumer.fetch.latency.avg\":{\"value\":0.39990625},\"kafka.consumer.assigned.partitions\":{\"value\":1.0},\"kafka.consumer.fetch.throttle.time.max\":{\"value\":0.0},\"kafka.consumer.fetch.throttle.time.avg\":{\"value\":0.0},\"kafka .consumer.join.time.avg\":{\"value\":0.0},\"kafka.consumer.io.wait.ratio\":{\"value\":0.998773698204943},\"kafka.consumer.sync.time.avg\":{\"value\":0.0},\"kafka.consumer.connection.count\":{\"value\":2.0},\"kafka.consumer.commit.latency.max\":{\"value\":0.001},\"kafka.consumer.io-wait-time-avg\":{\"value\":0.095 28972525378152},\"kafka.consumer.sync.rate\":{\"value\":0.0},\"kafka.consumer.join.rate\":{\"value\":0.0},\"kafka.consumer.fetch.total\":{\"value\":\"NaN\"},\"kafka.consumer.io.ratio\":{\"value\":0.00021523531276974297},\"kafka.consumer.heartbeat.response.time.max\":{\"value\":0.001},\"kafka.consumer.commit.rate\ ":{\"value\":0.21550175993103945},\"kafka.consumer.fetch.latency.max\":{\"value\":0.543}}}}"
        },                                                                                                                                                                                                                                                                                                     
        {                                                                                                                                                                                                                                                                                                      
          "message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/kafka.consumer.fetch.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string",
          "document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-2\"},\"samples\":{\"kafka.consumer.commit.latency.avg\":{\"value\":0.00045454545454545455},\"kafka.consumer.heartbeat.rate\":{\"value\":0.33676703645007927},\"kafka.consumer.io-time-avg\":{\"value\":3.75610205249816E-7},\"kafka.consumer.last-heartbeat\":{\"value\":2.0},\"kafka.consumer.fetch.latency.avg\":{\"value\":0.500897435897436},\"kafka.consumer.assigned.partitions\":{\"value\":1.0},\"kafka.consumer.fetch.throttle.time.max\":{\"value\":0.0},\"kafka.consumer.fetch.throttle.time.avg\":{\"value\":0.0},\"kafka.consumer.join.time.avg\":{\"value\":0.0},\"kafka.consumer.io.wait.ratio\":{\"value\":1.000531252284521},\"kafka.consumer.sync.time.avg\":{\"value\":0.0},\"kafka.consumer.commit.latency.max\":{\"value\":0.001},\"kafka.consumer.connection.count\":{\"value\":2.0},\"kafka.consumer.io-wait-time-avg\":{\"value\":0.0009365181971052416},\"kafka.consumer.sync.rate\":{\"value\":0.0},\"kafka.consumer.join.rate\":{\"value\":0.0},\"kafka.consumer.fetch.total\":{\"value\":\"NaN\"},\"kafka.consumer.io.ratio\":{\"value\":0.00040128397952230355},\"kafka.consumer.heartbeat.response.time.max\":{\"value\":0.001},\"kafka.consumer.commit.rate\":{\"value\":0.21395783085660935},\"kafka.consumer.fetch.latency.max\":{\"value\":0.502}}}}"
        },                                                                                                                                                                                                                                                                                                     
        {                                                                                                                                                                                                                                                                                                      
          "message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/kafka.consumer.records.consumed.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string",
          "document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-2\",\"topic\":\"dipp\"},\"samples\":{\"kafka.consumer.fetch.size.max\":{\"value\":0.0},\"kafka.consumer.records.per.request.avg\":{\"value\":0.0},\"kafka.consumer.bytes.consumed.total\":{\"value\":\"NaN\"},\"kafka.consumer.fetch.size.avg\":{\"value\":0.0},\"kafka.consumer.records.consumed.total\":{\"value\":\"NaN\"}}}}"
        }                                                                                                                                                                                                                                                                                                      
      ]                                                                                                                                                                                                                                                                                                        
    }

Elasticsearch version: 7.5.0

APM Server version: 7.5.2

APM Agent language and version: Java, 1.19.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Installed from official Elasticsearch repository

Fresh install or upgraded from other version? Fresh installation

Sylvain_Juge · December 3, 2020, 4:22pm

Thanks for reporting this @antonro.

Yes, since the agent version 1.18.1, you can use the disable_metrics option to disable those micrometer metrics.

We already had a similar bug in the micrometer metrics serialization in the past https://github.com/elastic/apm-agent-java/pull/1419

Can you provide us the Kafka driver and Micrometer libraries that you are using here ?

I'll open an new issue to track down this problem. In the mean time you can use configuration to avoid the errors.

antonro · December 3, 2020, 4:58pm

Hi @Sylvain_Juge ,

Thank you for the fast reply! Will definitely try using workaround and suppress kafka.consumer metrics collection.
We use following kafka and micrometer libraries in affected services:

org/apache/kafka/kafka-clients/0.11.0.0/kafka-clients-0.11.0.0.jar
org/springframework/kafka/spring-kafka/1.3.0.RELEASE/spring-kafka-1.3.0.RELEASE.jar
io/micrometer/micrometer-core/1.2.1/micrometer-core-1.2.1.jar

Sylvain_Juge · December 4, 2020, 12:55pm

Thanks, I've opened an issue in our repository so we can track it, feel free to subscribe to it to know when it's fixed.

Eyal_Koren · December 7, 2020, 7:27pm

@antonro Please try this snapshot (make sure you use the right artifact) and see if it fixes the problem without setting the disable_metrics config.

system · December 28, 2020, 3:27pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
APM Java Agent error APM java	9	2057	June 23, 2021
APM Java agnet return 400 sending report to elastic and create holes - co.elastic.apm.agent.report.IntakeV2ReportingEventHandler APM java	8	818	December 27, 2020
APM AGENT Issue Elasticsearch	2	822	February 13, 2020
Elastic Java APM Agent Fails with 400 Error Due to Traceparent Header in Kafka Messages APM docker , java , server	1	29	August 30, 2024
Java apm-agent http status codes not correctly identified APM	6	1498	October 18, 2018

kafka.consumer.bytes.consumed.total=NaN causes Error sending data to APM server: Server returned HTTP response code: 400

Related topics