# kafka.consumer.bytes.consumed.total=NaN causes Error sending data to APM server: Server returned HTTP response code: 400 **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [December 3, 2020, 4:02pm UTC](https://discuss.elastic.co/t/kafka-consumer-bytes-consumed-total-nan-causes-error-sending-data-to-apm-server-server-returned-http-response-code-400/257544 "2020-12-03T16:02:36Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![antonro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonro/32/41619_2.png) [@antonro](https://discuss.elastic.co/u/antonro)\ **Post date:** [December 3, 2020, 4:02pm UTC](https://discuss.elastic.co/t/kafka-consumer-bytes-consumed-total-nan-causes-error-sending-data-to-apm-server-server-returned-http-response-code-400/257544/1 "2020-12-03T16:02:36Z")
Hi, We started using Java APM agent and observe following errors in log files of our Java services with enabled agent. I see that some metrics are reported with NaN value which causes error with "expected number, but got string" description. Is there a way to resolve this or if not possible to suppress specific metric collection on agent side? Thanks! ```auto 2020-12-03 14:15:31,392 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%) 2020-12-03 14:15:31,392 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 400 for URL: http://localhost:8200/intake/v2/events, response code is 400 2020-12-03 14:15:31,392 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - { "accepted": 56, "errors": [ { "message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/kafka.consumer.bytes.consumed.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string", "document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-1\",\"topic\":\"scheduler\"},\"samples\":{\"kafka.consumer.records.per.request.avg\":{\"value\":0.2890625},\"kafka.consumer.fetch.size.max\":{\"value\":360.0},\"kafka.consumer.bytes.consumed.total\ ":{\"value\":\"NaN\"},\"kafka.consumer.records.consumed.total\":{\"value\":\"NaN\"},\"kafka.consumer.fetch.size.avg\":{\"value\":97.5234375}}}}" }, { "message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/kafka.consumer.fetch.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string", "document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-1\"},\"samples\":{\"kafka.consumer.commit.latency.avg\":{\"value\":0.0006666666666666666},\"kafka.consumer.heartbeat.rate\":{\"value\":0.3436819796082025},\"kafka.consumer.io-time-avg\":{\"value\": 0.00002053489579831933},\"kafka.consumer.last-heartbeat\":{\"value\":1.0},\"kafka.consumer.fetch.latency.avg\":{\"value\":0.39990625},\"kafka.consumer.assigned.partitions\":{\"value\":1.0},\"kafka.consumer.fetch.throttle.time.max\":{\"value\":0.0},\"kafka.consumer.fetch.throttle.time.avg\":{\"value\":0.0},\"kafka .consumer.join.time.avg\":{\"value\":0.0},\"kafka.consumer.io.wait.ratio\":{\"value\":0.998773698204943},\"kafka.consumer.sync.time.avg\":{\"value\":0.0},\"kafka.consumer.connection.count\":{\"value\":2.0},\"kafka.consumer.commit.latency.max\":{\"value\":0.001},\"kafka.consumer.io-wait-time-avg\":{\"value\":0.095 28972525378152},\"kafka.consumer.sync.rate\":{\"value\":0.0},\"kafka.consumer.join.rate\":{\"value\":0.0},\"kafka.consumer.fetch.total\":{\"value\":\"NaN\"},\"kafka.consumer.io.ratio\":{\"value\":0.00021523531276974297},\"kafka.consumer.heartbeat.response.time.max\":{\"value\":0.001},\"kafka.consumer.commit.rate\ ":{\"value\":0.21550175993103945},\"kafka.consumer.fetch.latency.max\":{\"value\":0.543}}}}" }, { "message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/kafka.consumer.fetch.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string", "document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-2\"},\"samples\":{\"kafka.consumer.commit.latency.avg\":{\"value\":0.00045454545454545455},\"kafka.consumer.heartbeat.rate\":{\"value\":0.33676703645007927},\"kafka.consumer.io-time-avg\":{\"value\":3.75610205249816E-7},\"kafka.consumer.last-heartbeat\":{\"value\":2.0},\"kafka.consumer.fetch.latency.avg\":{\"value\":0.500897435897436},\"kafka.consumer.assigned.partitions\":{\"value\":1.0},\"kafka.consumer.fetch.throttle.time.max\":{\"value\":0.0},\"kafka.consumer.fetch.throttle.time.avg\":{\"value\":0.0},\"kafka.consumer.join.time.avg\":{\"value\":0.0},\"kafka.consumer.io.wait.ratio\":{\"value\":1.000531252284521},\"kafka.consumer.sync.time.avg\":{\"value\":0.0},\"kafka.consumer.commit.latency.max\":{\"value\":0.001},\"kafka.consumer.connection.count\":{\"value\":2.0},\"kafka.consumer.io-wait-time-avg\":{\"value\":0.0009365181971052416},\"kafka.consumer.sync.rate\":{\"value\":0.0},\"kafka.consumer.join.rate\":{\"value\":0.0},\"kafka.consumer.fetch.total\":{\"value\":\"NaN\"},\"kafka.consumer.io.ratio\":{\"value\":0.00040128397952230355},\"kafka.consumer.heartbeat.response.time.max\":{\"value\":0.001},\"kafka.consumer.commit.rate\":{\"value\":0.21395783085660935},\"kafka.consumer.fetch.latency.max\":{\"value\":0.502}}}}" }, { "message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/kafka.consumer.records.consumed.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string", "document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-2\",\"topic\":\"dipp\"},\"samples\":{\"kafka.consumer.fetch.size.max\":{\"value\":0.0},\"kafka.consumer.records.per.request.avg\":{\"value\":0.0},\"kafka.consumer.bytes.consumed.total\":{\"value\":\"NaN\"},\"kafka.consumer.fetch.size.avg\":{\"value\":0.0},\"kafka.consumer.records.consumed.total\":{\"value\":\"NaN\"}}}}" } ] } ``` **Elasticsearch version** : 7.5.0 **APM Server version** : 7.5.2 **APM Agent language and version** : Java, 1.19.0 **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Installed from official Elasticsearch repository **Fresh install or upgraded from other version?** Fresh installation ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [December 3, 2020, 4:22pm UTC](https://discuss.elastic.co/t/kafka-consumer-bytes-consumed-total-nan-causes-error-sending-data-to-apm-server-server-returned-http-response-code-400/257544/2 "2020-12-03T16:22:59Z")
Thanks for reporting this @antonro. Yes, since the agent version [1.18.1](https://www.elastic.co/guide/en/apm/agent/java/master/release-notes-1.x.html#release-notes-1.18.1), you can use the `disable_metrics` option to disable those micrometer metrics. We already had a similar bug in the micrometer metrics serialization in the past [https://github.com/elastic/apm-agent-java/pull/1419](https://github.com/elastic/apm-agent-java/pull/1419) Can you provide us the Kafka driver and Micrometer libraries that you are using here ? I'll open an new issue to track down this problem. In the mean time you can use configuration to avoid the errors. ---
**Author:** ![antonro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonro/32/41619_2.png) [@antonro](https://discuss.elastic.co/u/antonro)\ **Post date:** [December 3, 2020, 4:58pm UTC](https://discuss.elastic.co/t/kafka-consumer-bytes-consumed-total-nan-causes-error-sending-data-to-apm-server-server-returned-http-response-code-400/257544/3 "2020-12-03T16:58:45Z")
Hi @Sylvain_Juge , Thank you for the fast reply! Will definitely try using workaround and suppress kafka.consumer metrics collection. We use following kafka and micrometer libraries in affected services: org/apache/kafka/kafka-clients/0.11.0.0/ **kafka-clients-0.11.0.0.jar** org/springframework/kafka/spring-kafka/1.3.0.RELEASE/ **spring-kafka-1.3.0.RELEASE.jar** io/micrometer/micrometer-core/1.2.1/ **micrometer-core-1.2.1.jar** ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [December 4, 2020, 12:55pm UTC](https://discuss.elastic.co/t/kafka-consumer-bytes-consumed-total-nan-causes-error-sending-data-to-apm-server-server-returned-http-response-code-400/257544/4 "2020-12-04T12:55:15Z")
Thanks, I've opened [an issue in our repository](https://github.com/elastic/apm-agent-java/issues/1554) so we can track it, feel free to subscribe to it to know when it's fixed. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [December 7, 2020, 7:27pm UTC](https://discuss.elastic.co/t/kafka-consumer-bytes-consumed-total-nan-causes-error-sending-data-to-apm-server-server-returned-http-response-code-400/257544/5 "2020-12-07T19:27:23Z")
@antonro Please try [this snapshot](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-java%2Fapm-agent-java-mbp/detail/PR-1559/2/artifacts) (make sure you use the right artifact) and see if it fixes the problem without setting the `disable_metrics` config. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 28, 2020, 3:27pm UTC](https://discuss.elastic.co/t/kafka-consumer-bytes-consumed-total-nan-causes-error-sending-data-to-apm-server-server-returned-http-response-code-400/257544/6 "2020-12-28T15:27:36Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.