Hi,
We started using Java APM agent and observe following errors in log files of our Java services with enabled agent. I see that some metrics are reported with NaN value which causes error with "expected number, but got string" description. Is there a way to resolve this or if not possible to suppress specific metric collection on agent side?
Thanks!
2020-12-03 14:15:31,392 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
2020-12-03 14:15:31,392 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 400 for URL: http://localhost:8200/intake/v2/events, response code is 400
2020-12-03 14:15:31,392 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {
"accepted": 56,
"errors": [
{
"message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/kafka.consumer.bytes.consumed.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string",
"document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-1\",\"topic\":\"scheduler\"},\"samples\":{\"kafka.consumer.records.per.request.avg\":{\"value\":0.2890625},\"kafka.consumer.fetch.size.max\":{\"value\":360.0},\"kafka.consumer.bytes.consumed.total\ ":{\"value\":\"NaN\"},\"kafka.consumer.records.consumed.total\":{\"value\":\"NaN\"},\"kafka.consumer.fetch.size.avg\":{\"value\":97.5234375}}}}"
},
{
"message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/kafka.consumer.fetch.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string",
"document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-1\"},\"samples\":{\"kafka.consumer.commit.latency.avg\":{\"value\":0.0006666666666666666},\"kafka.consumer.heartbeat.rate\":{\"value\":0.3436819796082025},\"kafka.consumer.io-time-avg\":{\"value\": 0.00002053489579831933},\"kafka.consumer.last-heartbeat\":{\"value\":1.0},\"kafka.consumer.fetch.latency.avg\":{\"value\":0.39990625},\"kafka.consumer.assigned.partitions\":{\"value\":1.0},\"kafka.consumer.fetch.throttle.time.max\":{\"value\":0.0},\"kafka.consumer.fetch.throttle.time.avg\":{\"value\":0.0},\"kafka .consumer.join.time.avg\":{\"value\":0.0},\"kafka.consumer.io.wait.ratio\":{\"value\":0.998773698204943},\"kafka.consumer.sync.time.avg\":{\"value\":0.0},\"kafka.consumer.connection.count\":{\"value\":2.0},\"kafka.consumer.commit.latency.max\":{\"value\":0.001},\"kafka.consumer.io-wait-time-avg\":{\"value\":0.095 28972525378152},\"kafka.consumer.sync.rate\":{\"value\":0.0},\"kafka.consumer.join.rate\":{\"value\":0.0},\"kafka.consumer.fetch.total\":{\"value\":\"NaN\"},\"kafka.consumer.io.ratio\":{\"value\":0.00021523531276974297},\"kafka.consumer.heartbeat.response.time.max\":{\"value\":0.001},\"kafka.consumer.commit.rate\ ":{\"value\":0.21550175993103945},\"kafka.consumer.fetch.latency.max\":{\"value\":0.543}}}}"
},
{
"message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/kafka.consumer.fetch.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string",
"document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-2\"},\"samples\":{\"kafka.consumer.commit.latency.avg\":{\"value\":0.00045454545454545455},\"kafka.consumer.heartbeat.rate\":{\"value\":0.33676703645007927},\"kafka.consumer.io-time-avg\":{\"value\":3.75610205249816E-7},\"kafka.consumer.last-heartbeat\":{\"value\":2.0},\"kafka.consumer.fetch.latency.avg\":{\"value\":0.500897435897436},\"kafka.consumer.assigned.partitions\":{\"value\":1.0},\"kafka.consumer.fetch.throttle.time.max\":{\"value\":0.0},\"kafka.consumer.fetch.throttle.time.avg\":{\"value\":0.0},\"kafka.consumer.join.time.avg\":{\"value\":0.0},\"kafka.consumer.io.wait.ratio\":{\"value\":1.000531252284521},\"kafka.consumer.sync.time.avg\":{\"value\":0.0},\"kafka.consumer.commit.latency.max\":{\"value\":0.001},\"kafka.consumer.connection.count\":{\"value\":2.0},\"kafka.consumer.io-wait-time-avg\":{\"value\":0.0009365181971052416},\"kafka.consumer.sync.rate\":{\"value\":0.0},\"kafka.consumer.join.rate\":{\"value\":0.0},\"kafka.consumer.fetch.total\":{\"value\":\"NaN\"},\"kafka.consumer.io.ratio\":{\"value\":0.00040128397952230355},\"kafka.consumer.heartbeat.response.time.max\":{\"value\":0.001},\"kafka.consumer.commit.rate\":{\"value\":0.21395783085660935},\"kafka.consumer.fetch.latency.max\":{\"value\":0.502}}}}"
},
{
"message": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/kafka.consumer.records.consumed.total/value] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/properties/value/type] expected number, but got string",
"document": "{\"metricset\":{\"timestamp\":1607004926799000,\"tags\":{\"client_id\":\"consumer-2\",\"topic\":\"dipp\"},\"samples\":{\"kafka.consumer.fetch.size.max\":{\"value\":0.0},\"kafka.consumer.records.per.request.avg\":{\"value\":0.0},\"kafka.consumer.bytes.consumed.total\":{\"value\":\"NaN\"},\"kafka.consumer.fetch.size.avg\":{\"value\":0.0},\"kafka.consumer.records.consumed.total\":{\"value\":\"NaN\"}}}}"
}
]
}
Elasticsearch version: 7.5.0
APM Server version: 7.5.2
APM Agent language and version: Java, 1.19.0
Original install method (e.g. download page, yum, deb, from source, etc.) and version: Installed from official Elasticsearch repository
Fresh install or upgraded from other version? Fresh installation