Elasticsearch version:
7.9.2 & tried 7.17.11
APM Server version:
7.10.2 & tried 7.17.11
APM Agent language and version:
elastic-apm-agent 1.42.0
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
added in pom.xml
Fresh install or upgraded from other version?
upgraded from 1.28.1
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
It seems there's a problem in the metrics data, I know I can disable all metrics as work a around but I want to see the metrics or some of it, how to go about this ? my other question how to extract the metric to block because when I set ELASTIC_APM_DISABLE_METRICS to 'http_server_requests,jvm_gc_pause' nothing changes.
Provide logs and/or server output (if relevant):
apm server logs:
ERROR [request] middleware/log_middleware.go:99 data validation error {"request_id": "2d690968-56ea-46fb-91e9-88354a220429", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.116.5.8", "user-agent": "apm-agent-java/1.42.0 (myapp)", "event.duration": 10019274911, "response_code": 400, "error": "failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/jvm_gc_pause.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\", failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/jvm_gc_pause.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\", failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/http_server_requests.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\""}```
springboot log
WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: {
"accepted": 84,
"errors": [
{
"message": "failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/jvm_gc_pause.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\"",
"document": "{\"metricset\":{\"timestamp\":1695023459832000,\"tags\":{\"action\":\"end of minor GC\",\"application\":\"myapp\",\"cause\":\"Allocation Failure\"},\"samples\":{\"jvm_gc_pause.count\":{\"value\":43},\"jvm_gc_pause.sum.us\":{\"value\":743000.0},\"jvm_gc_pause.histogram\":{\"values\":[],\"counts\":[]}}}}"
},
{
"message": "failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/jvm_gc_pause.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\"",
"document": "{\"metricset\":{\"timestamp\":1695023459832000,\"tags\":{\"action\":\"end of major GC\",\"application\":\"myapp\",\"cause\":\"Metadata GC Threshold\"},\"samples\":{\"jvm_gc_pause.count\":{\"value\":2},\"jvm_gc_pause.sum.us\":{\"value\":279000.0},\"jvm_gc_pause.histogram\":{\"values\":[],\"counts\":[]}}}}"
},
{
"message": "failed to validate metricset: error validating JSON: I[#] S[#] doesn't validate with \"metricset#\"\n I[#] S[#/allOf/5] allOf failed\n I[#/samples/http_server_requests.histogram] S[#/allOf/5/properties/samples/patternProperties/%5E%5B%5E%2A%22%5D%2A$/required] missing properties: \"value\"",
"document": "{\"metricset\":{\"timestamp\":1695023459832000,\"tags\":{\"application\":\"myapp\",\"exception\":\"None\",\"method\":\"GET\",\"outcome\":\"SUCCESS\",\"status\":\"200\",\"uri\":\"/management/health\"},\"samples\":{\"http_server_requests.count\":{\"value\":12},\"http_server_requests.sum.us\":{\"value\":3066128.122},\"http_server_requests.histogram\":{\"values\":[],\"counts\":[]}}}}"
}
]
}