Prometheus metrics coming in slowly

I'm not entirely sure what is causing this.

autodiscover is configured like so:

- type: kubernetes
  include_annotations: ["prometheus.io.scrape", "prometheus.io.port"]
  resource: service
  templates:
    - condition:
        and:
          - contains:
              kubernetes.annotations.prometheus.io/scrape: "true"
          - equals:
              kubernetes.service.name: "infinispan"
      config:
        - module: prometheus
          fields_under_root: true
          fields:
            event.dataset: prometheus.infinispan
          period: 20s
          hosts: ["${data.host}:${data.kubernetes.annotations.prometheus.io/port}"]
          metrics_path: /metrics
          metricsets: ["collector"]
    - condition:
        and:
          - contains:
              kubernetes.annotations.prometheus.io/scrape: "true"
          - not:
              equals:
                kubernetes.service.name: "infinispan"
      config:
        - module: prometheus
          period: 20s
          # Prometheus exporter host / port
          hosts: ["${data.host}:${data.port}"]
          metrics_path: /metrics
          metricsets: ["collector"]

nothing too crazy.

Metrics DO come in, but they are coming in anywhere from 20-25 mins apart. This is very much the case for our galera metrics:

Monitoring our databases is... very important as you can imagine.

Metricbeat logs reveal nothing out of the ordinary. I checked elasticsearch logs for any errors coming in, and it's pretty clean.

Is there something I have overlooked here? I've been scratching my head and pulling my hair out over this for the past 3 days.

ANY help would be very much appreciated.

Metrics DO come in, but they are coming in anywhere from 20-25 mins apart. This is very much the case for our galera metrics:

What do you mean by this? Do metrics just disappear or are not collected at all?

They do not disappear; they aren't being collected. As you can see in the picture I supplied in the OP, they come in every 20-25 mins or so.

The prometheus metrics are scheduled for 20 second intervals; but for some reason they just don't.

Metricbeat logs reveal nothing of relevance, nor does any logs from elasticsearch.

As a quick test, I ran a while loop targeting the prometheus metrics endpoint using curl to see if maybe it does not scrape everything on each interval, but every curl produced all the metrics, every time.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.