# Prometheus metrics coming in slowly

**URL:** <https://discuss.elastic.co/t/prometheus-metrics-coming-in-slowly/313921>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [September 7, 2022, 5:57pm UTC](https://discuss.elastic.co/t/prometheus-metrics-coming-in-slowly/313921 "2022-09-07T17:57:28Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Renato\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/renato_d/32/104415_2.png) [@Renato\_D](https://discuss.elastic.co/u/Renato_D)\
**Post date:** [September 7, 2022, 5:57pm UTC](https://discuss.elastic.co/t/prometheus-metrics-coming-in-slowly/313921/1 "2022-09-07T17:57:28Z")

</div>

I'm not entirely sure what is causing this.

autodiscover is configured like so:

```auto
- type: kubernetes
  include_annotations: ["prometheus.io.scrape", "prometheus.io.port"]
  resource: service
  templates:
    - condition:
        and:
          - contains:
              kubernetes.annotations.prometheus.io/scrape: "true"
          - equals:
              kubernetes.service.name: "infinispan"
      config:
        - module: prometheus
          fields_under_root: true
          fields:
            event.dataset: prometheus.infinispan
          period: 20s
          hosts: ["${data.host}:${data.kubernetes.annotations.prometheus.io/port}"]
          metrics_path: /metrics
          metricsets: ["collector"]
    - condition:
        and:
          - contains:
              kubernetes.annotations.prometheus.io/scrape: "true"
          - not:
              equals:
                kubernetes.service.name: "infinispan"
      config:
        - module: prometheus
          period: 20s
          # Prometheus exporter host / port
          hosts: ["${data.host}:${data.port}"]
          metrics_path: /metrics
          metricsets: ["collector"]

```

nothing too crazy.

Metrics DO come in, but they are coming in anywhere from 20-25 mins apart. This is very much the case for our galera metrics:

 ![kibana](https://us1.discourse-cdn.com/elastic/original/3X/d/7/d7561e2a0447a6fd93a3b2c4b2b3185c83035e73.jpeg)

Monitoring our databases is... very important as you can imagine.

Metricbeat logs reveal nothing out of the ordinary. I checked elasticsearch logs for any errors coming in, and it's pretty clean.

Is there something I have overlooked here? I've been scratching my head and pulling my hair out over this for the past 3 days.

ANY help would be very much appreciated.

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [September 8, 2022, 8:27am UTC](https://discuss.elastic.co/t/prometheus-metrics-coming-in-slowly/313921/2 "2022-09-08T08:27:26Z")

</div>

> Metrics DO come in, but they are coming in anywhere from 20-25 mins apart. This is very much the case for our galera metrics:

What do you mean by this? Do metrics just disappear or are not collected at all?

---

<div class="post-metadata">

**Author:** ![Renato\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/renato_d/32/104415_2.png) [@Renato\_D](https://discuss.elastic.co/u/Renato_D)\
**Post date:** [September 8, 2022, 1:47pm UTC](https://discuss.elastic.co/t/prometheus-metrics-coming-in-slowly/313921/3 "2022-09-08T13:47:05Z")

</div>

They do not disappear; they aren't being collected. As you can see in the picture I supplied in the OP, they come in every 20-25 mins or so.

The prometheus metrics are scheduled for 20 second intervals; but for some reason they just _don't_.

Metricbeat logs reveal nothing of relevance, nor does any logs from elasticsearch.

As a quick test, I ran a while loop targeting the prometheus metrics endpoint using curl to see if maybe it does not scrape everything on each interval, but every curl produced all the metrics, every time.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 6, 2022, 3:47pm UTC](https://discuss.elastic.co/t/prometheus-metrics-coming-in-slowly/313921/4 "2022-10-06T15:47:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.