Prometheus metrics coming in slowly

[Renato_D (Renato Dattilo)]

I'm not entirely sure what is causing this.

autodiscover is configured like so:

- type: kubernetes
  include_annotations: ["prometheus.io.scrape", "prometheus.io.port"]
  resource: service
  templates:
    - condition:
        and:
          - contains:
              kubernetes.annotations.prometheus.io/scrape: "true"
          - equals:
              kubernetes.service.name: "infinispan"
      config:
        - module: prometheus
          fields_under_root: true
          fields:
            event.dataset: prometheus.infinispan
          period: 20s
          hosts: ["${data.host}:${data.kubernetes.annotations.prometheus.io/port}"]
          metrics_path: /metrics
          metricsets: ["collector"]
    - condition:
        and:
          - contains:
              kubernetes.annotations.prometheus.io/scrape: "true"
          - not:
              equals:
                kubernetes.service.name: "infinispan"
      config:
        - module: prometheus
          period: 20s
          # Prometheus exporter host / port
          hosts: ["${data.host}:${data.port}"]
          metrics_path: /metrics
          metricsets: ["collector"]

nothing too crazy.

Metrics DO come in, but they are coming in anywhere from 20-25 mins apart. This is very much the case for our galera metrics:

Monitoring our databases is... very important as you can imagine.

Metricbeat logs reveal nothing out of the ordinary. I checked elasticsearch logs for any errors coming in, and it's pretty clean.

Is there something I have overlooked here? I've been scratching my head and pulling my hair out over this for the past 3 days.

ANY help would be very much appreciated.

[mtojek (Marcin Tojek)]

Metrics DO come in, but they are coming in anywhere from 20-25 mins apart. This is very much the case for our galera metrics:

What do you mean by this? Do metrics just disappear or are not collected at all?

[Renato_D (Renato Dattilo)]

They do not disappear; they aren't being collected. As you can see in the picture I supplied in the OP, they come in every 20-25 mins or so.

The prometheus metrics are scheduled for 20 second intervals; but for some reason they just don't.

Metricbeat logs reveal nothing of relevance, nor does any logs from elasticsearch.

As a quick test, I ran a while loop targeting the prometheus metrics endpoint using curl to see if maybe it does not scrape everything on each interval, but every curl produced all the metrics, every time.