# Elastic APM Agent and OpenTelemetry Agent Issue

**URL:** https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273
**Category:** APM
**Tags:** java
**Created:** [November 16, 2023, 5:00am UTC](https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273 "2023-11-16T05:00:25Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![baoletrg](https://avatars.discourse-cdn.com/v4/letter/b/c2a13f/32.png) [@baoletrg](https://discuss.elastic.co/u/baoletrg)
#### Post date: [November 16, 2023, 5:00am UTC](https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273/1 "2023-11-16T05:00:25Z")

</div>

I have implemented both APM Agent and OpenTelemetry Agent forward data to APM Server, but all the metrics data have this issues: No Pod metrics, No JVM metrics  
This seem relate to these field metricset.name, metricset.samples.name and metricset.samples.value, they wrapped on the metric data inside. Please see the attached images.  
Thank everyone.

 ![java_metrics](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fc3e974c02ae68dfe00487bb0264dc7ea2de92f4.png)  
 ![pod_metrics](https://us1.discourse-cdn.com/elastic/original/3X/4/3/432b8b3f71e7a0ebef568eab99956ffc54208e1e.png)  
 ![opentelemetry](https://us1.discourse-cdn.com/elastic/original/3X/1/2/12995a7ebdc44cdaa57e4f92847c996c62d54948.png)  
 ![apm-agent](https://us1.discourse-cdn.com/elastic/original/3X/9/e/9e433a55bd8510ce8f518e4c268ff6f2d29cfc7d.png)

---

<div class="post-metadata">

### Author: ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)
#### Post date: [November 16, 2023, 8:28am UTC](https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273/2 "2023-11-16T08:28:26Z")

</div>

Hi,

Does the issue happens with both OTel and Elastic APM Agents ? If so, that is quite likely an issue with the metrics ingestion, thus more server-side than on the agents.

For example, do you have any ingest pipeline configured for the metrics ?  
Also, does the traces and other elements in APM UI work as expected ?

---

<div class="post-metadata">

### Author: ![baoletrg](https://avatars.discourse-cdn.com/v4/letter/b/c2a13f/32.png) [@baoletrg](https://discuss.elastic.co/u/baoletrg)
#### Post date: [November 16, 2023, 8:45am UTC](https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273/3 "2023-11-16T08:45:04Z")

</div>

Thank you for reply.  
I have checked again. With Elastic apm agent, everything work fined because all the fields are not wrapped in metricset.samples field.  
But for OpenTelemetry agent, there are 2 issues I can see here:

1. It uses different field names (those start with process.runtime)
2. And those fields are wrapped inside the metricset.samples

Is there any version of APM and Elasticsearch work well with OpenTelemetry Java agent ?

---

<div class="post-metadata">

### Author: ![baoletrg](https://avatars.discourse-cdn.com/v4/letter/b/c2a13f/32.png) [@baoletrg](https://discuss.elastic.co/u/baoletrg)
#### Post date: [November 16, 2023, 8:47am UTC](https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273/4 "2023-11-16T08:47:07Z")

</div>

I did not change anything.  
I have updated everything to the newest version of the stack 8.11  
Also update the APM integration.  
Spans and Logs work well, but Metrics have problems

---

<div class="post-metadata">

### Author: ![baoletrg](https://avatars.discourse-cdn.com/v4/letter/b/c2a13f/32.png) [@baoletrg](https://discuss.elastic.co/u/baoletrg)
#### Post date: [November 16, 2023, 9:40am UTC](https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273/5 "2023-11-16T09:40:46Z")

</div>

Hi Mr. Sylvain,  
I have run the ingest pipeline metrics-apm.app-8.11.2-preview-1699883522 against the document from Otel agent. It failed, so it can not break the metricset.samples field.

---

<div class="post-metadata">

### Author: ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)
#### Post date: [November 16, 2023, 2:50pm UTC](https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273/6 "2023-11-16T14:50:20Z")

</div>

From your previous messages I understand that it works as expected with Elastic APM agent, is that correct ?

If that's the case, I would recommend sticking to that option if it works better for you. Otel data storage and querying improvements is something that we are actively working right now, thus this is expected to change in the near future.

---

<div class="post-metadata">

### Author: ![baoletrg](https://avatars.discourse-cdn.com/v4/letter/b/c2a13f/32.png) [@baoletrg](https://discuss.elastic.co/u/baoletrg)
#### Post date: [November 17, 2023, 1:49am UTC](https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273/7 "2023-11-17T01:49:42Z")

</div>

Hi Sylvain,  
There are many support libraries that Elastic APM does not have as OTEL.  
However, I have checked again, this issue still happens with Elastic APM Agent.  
I try to test the document with the ingest pipeline and it failed at the last part of the script:

```auto
      "error": {
        "root_cause": [
          {
            "type": "script_exception",
            "reason": "runtime error",
            "script_stack": [
              "for (sample in ctx.metricset.samples) {\n String ",
              " ^---- HERE"
            ],
            "script": "Map dynamic_templates = new HashMap(); ...",
            "lang": "painless",
            "position": {
              "offset": 67,
              "start": 39,
              "end": 88
            }
          }

```

This is the document I use to test:

```
"metricset": {
  "name": "app",
  "samples": {
    "name": [
      "jvm_classes_loaded",
      "jvm_classes_unloaded",
      "jvm_gc_live_data_size",
      "jvm_gc_max_data_size",
      "jvm_gc_memory_allocated",
      "jvm_gc_memory_promoted",
      "jvm_gc_overhead",
      "jvm_threads_daemon",
      "jvm_threads_live",
      "jvm_threads_peak",
      "jvm_threads_started",
      "process_cpu_usage",
      "process_files_max",
      "process_files_open",
      "process_start_time",
      "process_uptime",
      "system_cpu_count",
      "system_cpu_usage",
      "system_load_average_1m",
      "tomcat_sessions_active_current",
      "tomcat_sessions_active_max",
      "tomcat_sessions_alive_max",
      "tomcat_sessions_created",
      "tomcat_sessions_expired",
      "tomcat_sessions_rejected"
    ],
    "value": [
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      1,
      7,
      18,
      39,
      43,
      44,
      50,
      19698,
      94897,
      1048576,
      1206528,
      15886120,
      1698978617,
      10021634048,
      59413334168
    ]
  }
},

```

I think there is a problem with the apm ingest pipeline. This is the one I use now, nothing change, update to the latest version

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/b/4bf0413ede347c41026d55298167e0c4f45da77f.png)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 15, 2023, 1:50am UTC](https://discuss.elastic.co/t/elastic-apm-agent-and-opentelemetry-agent-issue/347273/8 "2023-12-15T01:50:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
