# OTEL - Prometheus receiver and missing metrics **URL:** \ **Category:** Elastic Observability\ **Tags:** otel\ **Created:** [March 30, 2025, 8:08pm UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563 "2025-03-30T20:08:17Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Steve\_Foster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_foster/32/125583_2.png) [@Steve\_Foster](https://discuss.elastic.co/u/Steve_Foster)\ **Post date:** [March 30, 2025, 8:08pm UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/1 "2025-03-30T20:08:17Z")
Hi folks, i'm using the `prometheusreceiver` in a Deployment to scrape our pods every 60s, the metrics are then sent via OTLP to another instance of the collector running as a DaemonSet, which then sends them onto elastic cloud. What I'm finding is that the metrics are not coming through at regular 60s intervals, For example if there are 3 pods, each minute between 0 and 3 records of a metric are indexed. which suggests that they are being dropped at some point. In the collector that is performing the scrapes I can see that they being performed regularly, so I'm starting to think that there is an issue in the gateway collector or the elastic cloud end. Is there a way to identify that they gateway is performing as expected? So that I can eliminate that part of the chain. [cross posted with CNCF slack #otel-collector] ---
**Author:** ![Steve\_Foster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_foster/32/125583_2.png) [@Steve\_Foster](https://discuss.elastic.co/u/Steve_Foster)\ **Post date:** [March 30, 2025, 8:45pm UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/2 "2025-03-30T20:45:42Z")
Ah looking at the internal metrics on the gateway collector I can see that the `otelcol_exporter_send_failed_metric_points` counter is increasing all the time. ---
**Author:** ![Steve\_Foster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_foster/32/125583_2.png) [@Steve\_Foster](https://discuss.elastic.co/u/Steve_Foster)\ **Post date:** [March 30, 2025, 8:50pm UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/3 "2025-03-30T20:50:04Z")
Possible root cause ```auto 2025-03-30T20:45:38.613Z error elasticsearchexporter@v0.112.0/bulkindexer.go:344 failed to index document {"kind": "exporter", "data_type": "metrics", "name": "elasticsearch/otel", "index": ".ds-metrics-otel-default-2025.03.30-000026", "error.type": "version_conflict_engine_exception", "error.reason": "[][@2025-03-30T20:45:21.786Z]: version conflict, document already exists (current version [1])", "hint": "check the \"Known issues\" section of Elasticsearch Exporter docs"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer ``` ---
**Author:** ![Steve\_Foster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_foster/32/125583_2.png) [@Steve\_Foster](https://discuss.elastic.co/u/Steve_Foster)\ **Post date:** [March 30, 2025, 8:57pm UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/4 "2025-03-30T20:57:41Z")
in the [known issues](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter#version_conflict_engine_exception) for the exporter there is a requirement to be on 8.16.5 or above. So our cluster is on that version, and even adding the component template as recommended for older versions doesn't address this. removing the batch processor as recommended doesn't appear to address this either· ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 31, 2025, 12:01am UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/5 "2025-03-31T00:01:48Z")
> [@Steve\_Foster](#): > > `.ds-metrics-otel-default-2025.03.30-000026` If you put in the component template it will not take affect until the data stream rolls over which could take some time You can try gong Kibana - Dev Tools `POST metrics-otel-default/_rollover` When you do that it will create the new backing index which use the component template...that said because TSDS the new metrics will not to start flow into that new backing index. For ~30 mins (long explanation left out) Plus you did share your configuration from either collector .. And I see you are using version 112.. I think the latest is 122 ---
**Author:** ![Steve\_Foster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_foster/32/125583_2.png) [@Steve\_Foster](https://discuss.elastic.co/u/Steve_Foster)\ **Post date:** [March 31, 2025, 2:17am UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/6 "2025-03-31T02:17:10Z")
> [@stephenb](#): > > If you put in the component template it will not take affect until the data stream rolls over which could take some time > > You can try gong Kibana - Dev Tools > > `POST metrics-otel-default/_rollover` > > When you do that it will create the new backing index which use the component template...that said because TSDS the new metrics will not to start flow into that new backing index. For ~30 mins (long explanation left out) Yes the index has rolled over already, the ILM policy that includes this data stream is set to roll over daily. > [@stephenb](#): > > Plus you did share your configuration from either collector .. In my support case I shared all 3 relevant configs used by the in-cluster operator. > [@stephenb](#): > > And I see you are using version 112.. I think the latest is 122 So I'm trying to follow the documented and supported versions... The generated values file that is passed into the helm chart sets the image to `docker.elastic.co/beats/elastic-agent:8.16.5`. Which matches the Elastic cluster version. Does the `otel/opentelemetry-collector-contrib:0.122.1` have the same distribution config as the elastic-agent? ---
**Author:** ![Steve\_Foster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_foster/32/125583_2.png) [@Steve\_Foster](https://discuss.elastic.co/u/Steve_Foster)\ **Post date:** [March 31, 2025, 2:30am UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/7 "2025-03-31T02:30:57Z")
Looks like 8.16.x elastic agent uses v0.112.0 versioned components from the opentelemetry distribution. > **[elastic-agent/internal/pkg/otel at v8.16.6 · elastic/elastic-agent](https://github.com/elastic/elastic-agent/tree/v8.16.6/internal/pkg/otel)** > > Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host. - elastic/elastic-agent In 8.17. this changes to v0.119.0 > **[elastic-agent/internal/pkg/otel at v8.17.4 · elastic/elastic-agent](https://github.com/elastic/elastic-agent/tree/v8.17.4/internal/pkg/otel)** > > Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host. - elastic/elastic-agent ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 31, 2025, 3:29am UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/8 "2025-03-31T03:29:46Z")
> [@Steve\_Foster](#): > > In my support case ... This is a public forum we do not have access to your support case, but good sounds like you are working with support ... > [@Steve\_Foster](#): > > In 8.17. this changes to v0.119.0 Might be worth a try... And of course as I am sure you know Elastic Agent in OTEL mode is still technical preview... > **[Run Elastic Agent as an OTel Collector | Fleet and Elastic Agent Guide \[8.17\]...](https://www.elastic.co/guide/en/fleet/current/otel-agent.html)** > This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features We absolutely appreciate you using / trying and reporting issues! ---
**Author:** ![Steve\_Foster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_foster/32/125583_2.png) [@Steve\_Foster](https://discuss.elastic.co/u/Steve_Foster)\ **Post date:** [March 31, 2025, 8:06am UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/9 "2025-03-31T08:06:03Z")
> [@stephenb](#): > > This is a public forum we do not have access to your support case, but good sounds like you are working with support ... happy to share here as well, is it best to just paste it in full or share a gist/pastebin? ---
**Author:** ![Steve\_Foster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_foster/32/125583_2.png) [@Steve\_Foster](https://discuss.elastic.co/u/Steve_Foster)\ **Post date:** [April 15, 2025, 8:59pm UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/10 "2025-04-15T20:59:00Z")
Just to follow up as I addressed this with the developer of the exporter > Using otel-colector-contrib:0.122.0 or otel-colector-contrib:0.124.0 I'm now seeing no errors about duplicate metrics. Our hosted Elastic cluster is running 8.16.6 > Ultimately we needed to add a component template using ```auto PUT _component_template/metrics-otel@custom { "template": { "mappings": { "properties": { "_metric_names_hash": { "type": "keyword", "time_series_dimension": true } } } } } ``` > Then roll over the data stream to use a new index (docs don't get routed into the new index for 30m) with ```auto POST metrics-prometheusreceiver.otel-default/_rollover ``` > Although the `_metric_names_hash` is indexed dynamically, i think it'll be missing the `time_series_dimension: true` on that field. > Also, i think you've helped us discover a bug in releasing this `_metric_names_hash` workaround. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 15, 2025, 9:35pm UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/11 "2025-04-15T21:35:41Z")
Nice @Steve_Foster Thank you ... Looking at these docs... > Was this the fix? > > > If metrics that have the same timestamp and dimensions aren't grouped into the s…ame document, ES will consider them to be a duplicate. This adds a hash of the metric names that will be mapped as a dimension in Elasticsearch. The tradeoff is that if the composition of the metrics grouping changes over time, a new time series will be created. That has an impact on the rate aggregation for counters. > > ES mapping changes: https://github.com/elastic/elasticsearch/pull/120952 ---
**Author:** ![Steve\_Foster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_foster/32/125583_2.png) [@Steve\_Foster](https://discuss.elastic.co/u/Steve_Foster)\ **Post date:** [April 15, 2025, 10:06pm UTC](https://discuss.elastic.co/t/otel-prometheus-receiver-and-missing-metrics/376563/12 "2025-04-15T22:06:06Z")
Yes however our hosted cluster is on 8.16.6 so shouldn't have needed the custom component template based on the wording of conditions about when you would need to add it or not. I think this is what foxed Carson in the first instance because it looked like it should have been fine.