400 failed to validate metricset: error validating JSON

Elasticsearch version:
7.9.2 & tried 7.17.11
APM Server version:
7.10.2 & tried 7.17.11

APM Agent language and version:
elastic-apm-agent 1.42.0
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
added in pom.xml

Fresh install or upgraded from other version?
upgraded from 1.28.1

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
It seems there's a problem in the metrics data, I know I can disable all metrics as work a around but I want to see the metrics or some of it, how to go about this ? my other question how to extract the metric to block because when I set ELASTIC_APM_DISABLE_METRICS to 'http_server_requests,jvm_gc_pause' nothing changes.

Provide logs and/or server output (if relevant):
apm server logs:

ERROR   [request]   middleware/log_middleware.go:99 data validation error   {"request_id": "2d690968-56ea-46fb-91e9-88354a220429", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.116.5.8", "user-agent": "apm-agent-java/1.42.0 (myapp)", "event.duration": 10019274911, "response_code": 400, "error": "failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/jvm_gc_pause.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\", failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/jvm_gc_pause.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\", failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/http_server_requests.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\""}```
springboot log
WARN  co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: {
  "accepted": 84,
  "errors": [
    {
      "message": "failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/jvm_gc_pause.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\"",
      "document": "{\"metricset\":{\"timestamp\":1695023459832000,\"tags\":{\"action\":\"end of minor GC\",\"application\":\"myapp\",\"cause\":\"Allocation Failure\"},\"samples\":{\"jvm_gc_pause.count\":{\"value\":43},\"jvm_gc_pause.sum.us\":{\"value\":743000.0},\"jvm_gc_pause.histogram\":{\"values\":[],\"counts\":[]}}}}"
    },
    {
      "message": "failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/jvm_gc_pause.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\"",
      "document": "{\"metricset\":{\"timestamp\":1695023459832000,\"tags\":{\"action\":\"end of major GC\",\"application\":\"myapp\",\"cause\":\"Metadata GC Threshold\"},\"samples\":{\"jvm_gc_pause.count\":{\"value\":2},\"jvm_gc_pause.sum.us\":{\"value\":279000.0},\"jvm_gc_pause.histogram\":{\"values\":[],\"counts\":[]}}}}"
    },
    {
      "message": "failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n  I[#] S[#/allOf/5] allOf failed\n    I[#/samples/http_server_requests.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\"",
      "document": "{\"metricset\":{\"timestamp\":1695023459832000,\"tags\":{\"application\":\"myapp\",\"exception\":\"None\",\"method\":\"GET\",\"outcome\":\"SUCCESS\",\"status\":\"200\",\"uri\":\"/management/health\"},\"samples\":{\"http_server_requests.count\":{\"value\":12},\"http_server_requests.sum.us\":{\"value\":3066128.122},\"http_server_requests.histogram\":{\"values\":[],\"counts\":[]}}}}"
    }
  ]
}

The bug is fixed in the current main branch and will be included in the next release of the agent (expected this week). The ELASTIC_APM_DISABLE_METRICS config applies to the metric name as produced by the JVM/application rather than the de-dotted name exposed in the UI, so for example setting to "jvm.memory.non_heap.used" would eliminate just that metric

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.