Elastic APM Agent and OpenTelemetry Agent Issue

[baoletrg (Bao Le Trong)]

I have implemented both APM Agent and OpenTelemetry Agent forward data to APM Server, but all the metrics data have this issues: No Pod metrics, No JVM metrics
This seem relate to these field metricset.name, metricset.samples.name and metricset.samples.value, they wrapped on the metric data inside. Please see the attached images.
Thank everyone.

[Sylvain_Juge]

Hi,

Does the issue happens with both OTel and Elastic APM Agents ? If so, that is quite likely an issue with the metrics ingestion, thus more server-side than on the agents.

For example, do you have any ingest pipeline configured for the metrics ?
Also, does the traces and other elements in APM UI work as expected ?

[baoletrg (Bao Le Trong)]

Thank you for reply.
I have checked again. With Elastic apm agent, everything work fined because all the fields are not wrapped in metricset.samples field.
But for OpenTelemetry agent, there are 2 issues I can see here:

1. It uses different field names (those start with process.runtime)
2. And those fields are wrapped inside the metricset.samples

Is there any version of APM and Elasticsearch work well with OpenTelemetry Java agent ?

[baoletrg (Bao Le Trong)]

I did not change anything.
I have updated everything to the newest version of the stack 8.11
Also update the APM integration.
Spans and Logs work well, but Metrics have problems

[baoletrg (Bao Le Trong)]

Hi Mr. Sylvain,
I have run the ingest pipeline metrics-apm.app-8.11.2-preview-1699883522 against the document from Otel agent. It failed, so it can not break the metricset.samples field.

[Sylvain_Juge]

From your previous messages I understand that it works as expected with Elastic APM agent, is that correct ?

If that's the case, I would recommend sticking to that option if it works better for you. Otel data storage and querying improvements is something that we are actively working right now, thus this is expected to change in the near future.

[baoletrg (Bao Le Trong)]

Hi Sylvain,
There are many support libraries that Elastic APM does not have as OTEL.
However, I have checked again, this issue still happens with Elastic APM Agent.
I try to test the document with the ingest pipeline and it failed at the last part of the script:

      "error": {
        "root_cause": [
          {
            "type": "script_exception",
            "reason": "runtime error",
            "script_stack": [
              "for (sample in ctx.metricset.samples) {\n  String ",
              "                            ^---- HERE"
            ],
            "script": "Map dynamic_templates = new HashMap(); ...",
            "lang": "painless",
            "position": {
              "offset": 67,
              "start": 39,
              "end": 88
            }
          }

This is the document I use to test:

"metricset": {
  "name": "app",
  "samples": {
    "name": [
      "jvm_classes_loaded",
      "jvm_classes_unloaded",
      "jvm_gc_live_data_size",
      "jvm_gc_max_data_size",
      "jvm_gc_memory_allocated",
      "jvm_gc_memory_promoted",
      "jvm_gc_overhead",
      "jvm_threads_daemon",
      "jvm_threads_live",
      "jvm_threads_peak",
      "jvm_threads_started",
      "process_cpu_usage",
      "process_files_max",
      "process_files_open",
      "process_start_time",
      "process_uptime",
      "system_cpu_count",
      "system_cpu_usage",
      "system_load_average_1m",
      "tomcat_sessions_active_current",
      "tomcat_sessions_active_max",
      "tomcat_sessions_alive_max",
      "tomcat_sessions_created",
      "tomcat_sessions_expired",
      "tomcat_sessions_rejected"
    ],
    "value": [
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      1,
      7,
      18,
      39,
      43,
      44,
      50,
      19698,
      94897,
      1048576,
      1206528,
      15886120,
      1698978617,
      10021634048,
      59413334168
    ]
  }
},

I think there is a problem with the apm ingest pipeline. This is the one I use now, nothing change, update to the latest version