# OpenTelemetry metrics don't have labels

**URL:** <https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510>\
**Category:** APM\
**Tags:** server, open-telemetry\
**Created:** [October 24, 2021, 8:55am UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510 "2021-10-24T08:55:17Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![benbek](https://avatars.discourse-cdn.com/v4/letter/b/ce73a5/32.png) [@benbek](https://discuss.elastic.co/u/benbek)\
**Post date:** [October 24, 2021, 8:55am UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/1 "2021-10-24T08:55:17Z")

</div>

Hello,

I am trying to understand why otel metrics don't display the labels I set in my code.

I use:

- go 1.17
- elastic/apm 7.15.1
- opentelemetry 1.0.1
- native otlp collector 0.6.0

My metrics index has:

```json
"apm-7.15.1-metric-000001" : {
    "mappings" : {
      "_meta" : {
        "beat" : "apm",
        "version" : "7.15.1"
      },
      "dynamic_templates" : [
        {
          "labels" : {
            "path_match" : "labels.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },

```

My golang code for adding one to a counter and attaching a label each time the value is incremented:

```auto
       warnMeter.warningCounter.Add(
		warnMeter.ctx,
		1,
		attribute.Int("labels.code", code),
		attribute.String("labels.text", text),
		attribute.String("labels.server", warnMeter.server),
	)

```

As you can see, I am trying to add one integer label and two string labels (or attributes, as the otel spec goes). Alas, I do not see any them in the index record (but I do see the counter value).

Considering a non-zero counter was only resolved on 7.15 (see the [apm-server issue](https://github.com/elastic/apm-server/issues/6185)), is there a chance otel metric labels are not yet supported? Or am I doing something wrong?

Thanks in advance!  
Ben

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [October 26, 2021, 7:02am UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/2 "2021-10-26T07:02:35Z")

</div>

Hi @benbek, welcome to the forum!

> As you can see, I am trying to add one integer label and two string labels (or attributes, as the otel spec goes). Alas, I do not see any them in the index record (but I do see the counter value).

Do you see _any_ labels? I created a test program with your code snippet, and I end up with a metric document with the following label fields:

```json
          "labels" : {
            "labels_server" : "server_name",
            "labels_text" : "some_text",
            "labels_code" : 400
          },

```

As you can see, "label" is repeated.

> [@benbek](#):
>
> ```auto
> attribute.Int("labels.code", code),
> attribute.String("labels.text", text),
> attribute.String("labels.server", warnMeter.server),
> 
> ```

"labels." will be automatically prefixed to any attributes which are not OTel semantic conventions known to apm-server. So you should change "labels.code" to just "code", for example.

---

<div class="post-metadata">

**Author:** ![benbek](https://avatars.discourse-cdn.com/v4/letter/b/ce73a5/32.png) [@benbek](https://discuss.elastic.co/u/benbek)\
**Post date:** [October 26, 2021, 11:55am UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/3 "2021-10-26T11:55:18Z")

</div>

Hi @axw, it's good to be here – and thank you for the prompt reply 🤗

I've corrected my code to not include the `labels.` prefix – but it still doesn't solve the problem. As a matter of fact, I don't see any labels in the record.

I've also created a separate go application that demonstrates the problem. Will be happy to share it with you. I will really appreciate if you could jump on a quick call for troubleshooting.

---

<div class="post-metadata">

**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\
**Post date:** [October 26, 2021, 1:43pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/4 "2021-10-26T13:43:55Z")

</div>

@benbek,

Could you try to attach the labels using the `Bind()` function? In the example below, I create an Int64Counter with an attribute and a description, which works fine.

> <https://github.com/riferrei/otel-with-golang/blob/50275c6aba9e911c2ba1fa1b753a04c81e096a75/main.go#L155>

This is what I get into the index:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b692426997b6e35de1e112b9aa47f8e47c56f55e.png)

For reference, I am using APM **7.15.1** here.

— @riferrei

---

<div class="post-metadata">

**Author:** ![benbek](https://avatars.discourse-cdn.com/v4/letter/b/ce73a5/32.png) [@benbek](https://discuss.elastic.co/u/benbek)\
**Post date:** [October 26, 2021, 2:18pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/5 "2021-10-26T14:18:16Z")

</div>

Hi @riferrei, unfortunately `Bind` also doesn't work for me – no labels are present. Actually, in our experience Int64Counters don't log anything in the latest version, and we had to switch to `Float64Counter`s.  
Anyway, `Bind()` with labels doesn't work.

This is the entire index record, for your reference:

```json
{
  "_index": "apm-7.15.1-metric-000001",
  "_type": "_doc",
  "_id": "YeLlvHwBt11rSkEJjx_g",
  "_version": 1,
  "_score": 1,
  "_source": {
    "agent": {
      "name": "opentelemetry/go",
      "version": "1.0.1"
    },
    "integer.counter": 1,
    "processor": {
      "name": "metric",
      "event": "metric"
    },
    "seconds.counter": 5,
    "metricset.name": "app",
    "observer": {
      "hostname": "8243f92668ee",
      "name": "instance-0000000005",
      "id": "c684a7b5-cabe-4038-8252-5fe7de80e56e",
      "ephemeral_id": "9af5300d-95b9-4447-a545-02b9ecc0d5f8",
      "type": "apm-server",
      "version": "7.15.1",
      "version_major": 7
    },
    "@timestamp": "2021-10-26T13:58:51.966Z",
    "_metric_descriptions": {
      "integer.counter": {
        "type": "counter"
      },
      "seconds.counter": {
        "type": "counter"
      }
    },
    "ecs": {
      "version": "1.11.0"
    },
    "service": {
      "name": "unknown_service ____ 1go_build_otel_example",
      "language": {
        "name": "go"
      }
    },
    "event": {
      "ingested": "2021-10-26T13:58:53.663301490Z"
    }
  },
  "fields": {
    "service.name": [
      "unknown_service ____ 1go_build_otel_example"
    ],
    "observer.name": [
      "instance-0000000005"
    ],
    "integer.counter": [
      1
    ],
    "processor.name": [
      "metric"
    ],
    "_metric_descriptions.integer.counter.type": [
      "counter"
    ],
    "observer.version_major": [
      7
    ],
    "service.language.name": [
      "go"
    ],
    "observer.hostname": [
      "8243f92668ee"
    ],
    "seconds.counter": [
      5
    ],
    "metricset.name": [
      "app"
    ],
    "observer.id": [
      "c684a7b5-cabe-4038-8252-5fe7de80e56e"
    ],
    "_metric_descriptions.seconds.counter.type": [
      "counter"
    ],
    "event.ingested": [
      "2021-10-26T13:58:53.663Z"
    ],
    "@timestamp": [
      "2021-10-26T13:58:51.966Z"
    ],
    "observer.ephemeral_id": [
      "9af5300d-95b9-4447-a545-02b9ecc0d5f8"
    ],
    "observer.version": [
      "7.15.1"
    ],
    "ecs.version": [
      "1.11.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "processor.event": [
      "metric"
    ],
    "agent.name": [
      "opentelemetry/go"
    ],
    "agent.version": [
      "1.0.1"
    ]
  }
}

```

(the bound metric is called `integer.counter` although it is Float64 😄 )

I'll be happy to provide any more details on the matter.

Thanks,  
Ben

---

<div class="post-metadata">

**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\
**Post date:** [October 26, 2021, 2:40pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/6 "2021-10-26T14:40:12Z")

</div>

Yup, that's a bummer 😅

At this point, the best thing to do is to analyze the problem at a code and dependencies level. If you could share a sample code that demonstrates the problem — I'm glad to take a look along with @axw 👍🏻

— @riferrei

---

<div class="post-metadata">

**Author:** ![benbek](https://avatars.discourse-cdn.com/v4/letter/b/ce73a5/32.png) [@benbek](https://discuss.elastic.co/u/benbek)\
**Post date:** [October 26, 2021, 2:55pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/7 "2021-10-26T14:55:22Z")

</div>

Cool!

Please see this gist: [OpenTelemetry + APM Metrics issue · GitHub](https://gist.github.com/benbek/1375cd7c5f217361d1ab0b5f357ca9a6)

There's only one thing to configure there, and that's [the endpoint in line 18](https://gist.github.com/benbek/1375cd7c5f217361d1ab0b5f357ca9a6#file-main-go-L18).

Appreciate you taking a look 👀

---

<div class="post-metadata">

**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\
**Post date:** [October 26, 2021, 3:10pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/8 "2021-10-26T15:10:43Z")

</div>

Just executed your code AS-IS and the labels are being correctly created:

```auto
{
  "_index": "apm-7.15.1-metric-000001",
  "_type": "_doc",
  "_id": "qVMlvXwBz4zw3S0yejZf",
  "_version": 1,
  "_score": 1,
  "_source": {
    "agent": {
      "name": "opentelemetry/go",
      "version": "1.0.1"
    },
    "processor": {
      "name": "metric",
      "event": "metric"
    },
    "seconds.counter": 105,
    "labels": {
      "server": "localhost",
      "code": 299,
      "text": "my text"
    },
    "metricset.name": "app",
    "observer": {
      "hostname": "apm-server",
      "id": "9fe95d0b-f2b9-4377-98de-09a8c16d2301",
      "type": "apm-server",
      "ephemeral_id": "bf3b6a10-1d7e-40db-b542-c6555c5c2885",
      "version": "7.15.1",
      "version_major": 7
    },
    "@timestamp": "2021-10-26T15:08:41.458Z",
    "ecs": {
      "version": "1.11.0"
    },
    "service": {
      "name": "unknown_service_main",
      "language": {
        "name": "go"
      }
    },
    "event": {
      "ingested": "2021-10-26T15:08:42.462222Z"
    }
  },
  "fields": {
    "service.name": [
      "unknown_service_main"
    ],
    "processor.name": [
      "metric"
    ],
    "labels.text": [
      "my text"
    ],
    "observer.version_major": [
      7
    ],
    "service.language.name": [
      "go"
    ],
    "observer.hostname": [
      "apm-server"
    ],
    "seconds.counter": [
      105
    ],
    "metricset.name": [
      "app"
    ],
    "labels.code": [
      299
    ],
    "labels.server": [
      "localhost"
    ],
    "observer.id": [
      "9fe95d0b-f2b9-4377-98de-09a8c16d2301"
    ],
    "event.ingested": [
      "2021-10-26T15:08:42.462Z"
    ],
    "@timestamp": [
      "2021-10-26T15:08:41.458Z"
    ],
    "observer.ephemeral_id": [
      "bf3b6a10-1d7e-40db-b542-c6555c5c2885"
    ],
    "observer.version": [
      "7.15.1"
    ],
    "ecs.version": [
      "1.11.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "processor.event": [
      "metric"
    ],
    "agent.name": [
      "opentelemetry/go"
    ],
    "agent.version": [
      "1.0.1"
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![benbek](https://avatars.discourse-cdn.com/v4/letter/b/ce73a5/32.png) [@benbek](https://discuss.elastic.co/u/benbek)\
**Post date:** [October 26, 2021, 4:49pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/9 "2021-10-26T16:49:17Z")

</div>

This is getting more and more interesting. If not the code, I guess the culprit lies somewhere in the ES cloud. How do you suggest we proceed? Should I open a support ticket so you guys could access it?

---

<div class="post-metadata">

**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\
**Post date:** [October 26, 2021, 4:56pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/10 "2021-10-26T16:56:13Z")

</div>

I tested this with Elastic APM running locally via Docker. I can test using Elastic Cloud. Just a sec...

---

<div class="post-metadata">

**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\
**Post date:** [October 26, 2021, 5:28pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/11 "2021-10-26T17:28:58Z")

</div>

Just validated here that — using the same code you provided — it also correctly created the labels in Elastic Cloud, as you can see here:

```auto
{
  "_index": "apm-7.15.1-metric-000001",
  "_type": "_doc",
  "_id": "8BGivXwBJ7UxgUg2gMtC",
  "_version": 1,
  "_score": 1,
  "_source": {
    "agent": {
      "name": "opentelemetry/go",
      "version": "1.0.1"
    },
    "processor": {
      "name": "metric",
      "event": "metric"
    },
    "seconds.counter": 10,
    "labels": {
      "server": "localhost",
      "code": 299,
      "text": "my text"
    },
    "metricset.name": "app",
    "observer": {
      "hostname": "ee23a1d26beb",
      "name": "instance-0000000000",
      "id": "36ed24d2-6502-4e97-bd09-bd744e122136",
      "ephemeral_id": "957ee57f-65b6-4ffd-a4e6-f32875a27b72",
      "type": "apm-server",
      "version": "7.15.1",
      "version_major": 7
    },
    "@timestamp": "2021-10-26T17:25:14.821Z",
    "ecs": {
      "version": "1.11.0"
    },
    "service": {
      "name": "unknown_service_main",
      "language": {
        "name": "go"
      }
    },
    "event": {
      "ingested": "2021-10-26T17:25:15.967759904Z"
    }
  },
  "fields": {
    "service.name": [
      "unknown_service_main"
    ],
    "observer.name": [
      "instance-0000000000"
    ],
    "processor.name": [
      "metric"
    ],
    "labels.text": [
      "my text"
    ],
    "observer.version_major": [
      7
    ],
    "service.language.name": [
      "go"
    ],
    "observer.hostname": [
      "ee23a1d26beb"
    ],
    "seconds.counter": [
      10
    ],
    "metricset.name": [
      "app"
    ],
    "labels.code": [
      299
    ],
    "labels.server": [
      "localhost"
    ],
    "observer.id": [
      "36ed24d2-6502-4e97-bd09-bd744e122136"
    ],
    "event.ingested": [
      "2021-10-26T17:25:15.967Z"
    ],
    "@timestamp": [
      "2021-10-26T17:25:14.821Z"
    ],
    "observer.ephemeral_id": [
      "957ee57f-65b6-4ffd-a4e6-f32875a27b72"
    ],
    "observer.version": [
      "7.15.1"
    ],
    "ecs.version": [
      "1.11.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "processor.event": [
      "metric"
    ],
    "agent.name": [
      "opentelemetry/go"
    ],
    "agent.version": [
      "1.0.1"
    ]
  }
}

```

But to use Elastic Cloud, I had to change your code to provide the bearer token for authentication.

```auto
type TelConfig struct {
	Endpoint string
	BearerToken string
}

```

⬇

```auto
securityDialOption := otlpmetricgrpc.WithInsecure()

var headersOption otlpmetricgrpc.Option

if strings.HasSuffix(t.config.Endpoint, ":443") {

securityDialOption = otlpmetricgrpc.WithTLSCredentials(credentials.NewTLS(&tls.Config{}))

headersOption = otlpmetricgrpc.WithHeaders(map[string]string{"Authorization": t.config.BearerToken})

}

```

— @riferrei

---

<div class="post-metadata">

**Author:** ![benbek](https://avatars.discourse-cdn.com/v4/letter/b/ce73a5/32.png) [@benbek](https://discuss.elastic.co/u/benbek)\
**Post date:** [October 26, 2021, 8:42pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/12 "2021-10-26T20:42:06Z")

</div>

Thanks for checking! Really appreciate it.  
By the process of elimination, could the issue be the otel collector? As mentioned, we use version 0.6.0 ([GitHub - open-telemetry/opentelemetry-helm-charts: OpenTelemetry Helm Charts](https://github.com/open-telemetry/opentelemetry-helm-charts)). Is there a possibility to do a re-test with that collector?

---

<div class="post-metadata">

**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\
**Post date:** [October 26, 2021, 8:48pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/13 "2021-10-26T20:48:56Z")

</div>

You can certainly test this _without_ the collector. Since your code has been written to send the metrics over OTel/gRPC and Elastic APM does support this natively — you can bypass the collector altogether.

TL;DR: your endpoint should be Elastic APM and not the collector.

— @riferrei

---

<div class="post-metadata">

**Author:** ![benbek](https://avatars.discourse-cdn.com/v4/letter/b/ce73a5/32.png) [@benbek](https://discuss.elastic.co/u/benbek)\
**Post date:** [October 26, 2021, 10:33pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/14 "2021-10-26T22:33:55Z")

</div>

Just confirmed it – I have labels when I'm talking directly to APM. Hooray!  
Created an issue for opentelemetry-collector: [Meter values sent to Elastic APM are missing labels · Issue #4273 · open-telemetry/opentelemetry-collector · GitHub](https://github.com/open-telemetry/opentelemetry-collector/issues/4273).

Once again, thank you for all your help 💟

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 23, 2021, 10:34pm UTC](https://discuss.elastic.co/t/opentelemetry-metrics-dont-have-labels/287510/15 "2021-11-23T22:34:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.