Errors in APM - requires at least one of the fields 'value;values'

Elastic Observability APM
,
1

Hi,

I'm trying to install a new ECK in my Kubernetes cluster. We are using the same Java APM agents from the older ELK, that worked properly.
We have the same Elastic properties in all Java APM agents, but some validation error: metricset: samples: requires at least one of the fields 'value;values' are showing.
The configs are:

image
image718×187 18 KB

I didn't have changed any config in Java Agents, and some agents work, some don't, not showing the traces.
image
image1668×758 79 KB

Can anyone help me to fix this issue?

Kibana version:
8.16.0
Elasticsearch version:
8.16.0
APM Server version:
8.16.0
APM Agent language and version:
Many versions, but above 1.x
Browser version:

Version 1.73.91 Chromium: 131.0.6778.85 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Helm ECK
Fresh install or upgraded from other version?
Fresh install
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
We had an old 8.0 Elastic, using an Elastic Java Agent. We've deployed a fresh new ECK in Kubernetes

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:

  1. Install ECK with Helm
  2. Add Elastic Java Agent to an application
  3. Check errors in APM

Provide logs and/or server output (if relevant):

apm-apm-server-84d4f958dc-kpfdg apm-server {"log.level":"error","@timestamp":"2024-12-03T18:33:15.275Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":59},"message":"data validation error","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.36.0 (bff-tablet 1.32.0)","source.address":"10.0.144.56","http.request.id":"2985c08f-9e06-4fb1-8c2f-312287cf05f6","event.duration":3288338,"http.request.body.bytes":2913,"http.response.status_code":400,"error.message":"validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values'","ecs.version":"1.6.0"}
apm-apm-server-84d4f958dc-kpfdg apm-server {"log.level":"error","@timestamp":"2024-12-03T18:33:16.270Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":59},"message":"data validation error","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.36.0 (quick-search-composite 2.20.4)","source.address":"192.168.99.194","http.request.id":"a97286eb-1272-49c2-a47f-1228ba9fce0c","event.duration":6359212,"http.request.body.bytes":5679,"http.response.status_code":400,"error.message":"validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values'","ecs.version":"1.6.0"}
apm-apm-server-84d4f958dc-qfb2p apm-server {"log.level":"error","@timestamp":"2024-12-03T18:33:16.404Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":59},"message":"data validation error","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.36.0 (ura-app 1.9.2)","source.address":"10.0.144.52","http.request.id":"4f1e1bf6-fe19-46b2-8890-9b2ba031c662","event.duration":2861353,"http.request.body.bytes":2844,"http.response.status_code":400,"error.message":"validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values', validation error: metricset: samples: requires at least one of the fields 'value;values'","ecs.version":"1.6.0"}
2

have you tried the latest java agent (1.52.1)?

3

Yes, but for some reason some applications report these invalid 'metricsets'

[INFO] +- co.elastic.apm:apm-agent-attach:jar:1.52.1:provided
[INFO] |  +- net.java.dev.jna:jna:jar:4.2.2:provided
[INFO] |  \- net.java.dev.jna:jna-platform:jar:4.2.2:provided
[INFO] +- co.elastic.apm:elastic-apm-agent:jar:1.52.1:compile

2024-12-04 11:42:41,831 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: {
  "accepted": 34,
  "errors": [
    {
      "message": "validation error: metricset: samples: requires at least one of the fields 'value;values'",
      "document": "{\"metricset\":{\"timestamp\":1733323359785000,\"tags\":{\"action\":\"end of minor GC\",\"cause\":\"Allocation Failure\"},\"samples\":{\"jvm_gc_pause.count\":{\"value\":9},\"jvm_gc_pause.sum.us\":{\"value\":392000.0},\"jvm_gc_pause.histogram\":{\"values\":[],\"counts\":[]}}}}"
    },
    {
      "message": "validation error: metricset: samples: requires at least one of the fields 'value;values'",
      "document": "{\"metricset\":{\"timestamp\":1733323359785000,\"tags\":{\"action\":\"end of major GC\",\"cause\":\"Metadata GC Threshold\"},\"samples\":{\"jvm_gc_pause.count\":{\"value\":2},\"jvm_gc_pause.sum.us\":{\"value\":814000.0},\"jvm_gc_pause.histogram\":{\"values\":[],\"counts\":[]}}}}"
    },
    {
      "message": "validation error: metricset: samples: requires at least one of the fields 'value;values'",
      "document": "{\"metricset\":{\"timestamp\":1733323359785000,\"tags\":{\"action\":\"end of minor GC\",\"cause\":\"Metadata GC Threshold\"},\"samples\":{\"jvm_gc_pause.count\":{\"value\":2},\"jvm_gc_pause.sum.us\":{\"value\":81000.0},\"jvm_gc_pause.histogram\":{\"values\":[],\"counts\":[]}}}}"
    }
  ]
}

Some worked, but some doesn't. I'll try to figure it out what is the difference.