[Metricbeat][Prometheus] Metricbeat 7.7 rate not stored when use_types and rate_counters option is set to true

Elastic Stack Beats
1

Hi all,

I am trying out the Metricbeat Prometheus collector module with the two new options in 7.7; use_types and rate_counters; https://www.elastic.co/guide/en/beats/metricbeat/7.7/metricbeat-metricset-prometheus-collector.html

While the Histogram metrics are stored in ElasticSearch as documented, the Counter information is not. Namely, there is no separate rate field.

Here is what an indexed metric looks like:

{
  "_index": "metricbeat-7.7.1-2020.06.08-000001",
  "_type": "_doc",
  "_id": "vzqzlnIB-vP0G_z2A1K5",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-06-09T01:30:12.793Z",
    "prometheus": {
      "method_duration": {
        "histogram": {
          "counts": [
            1,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0
          ],
          "values": [
            0.0025,
            0.0075,
            0.0175,
            0.037500000000000006,
            0.0625,
            0.0875,
            0.175,
            0.375,
            0.625,
            0.875,
            1.75,
            3.75,
            6.25,
            8.75,
            12.5
          ]
        }
      },
      "method_call_total": {
        "value": 161
      },
      "labels": {
        "job": "prometheus",
        "method_name": "run",
        "instance": "prometheus-test-service:8081"
      },
      "method_duration_created": {
        "value": 1591665413.6988633
      },
      "method_call_created": {
        "value": 1591665413.6987088
      }
    },
    "metricset": {
      "period": 5000,
      "name": "collector"
    },
    "service": {
      "type": "prometheus",
      "address": "prometheus-test-service:8081"
    },
    "event": {
      "module": "prometheus",
      "duration": 14211000,
      "dataset": "prometheus.collector"
    },
    "ecs": {
      "version": "1.5.0"
    },
    "host": {
      "name": "d0ab35b86d7a"
    },
    "agent": {
      "version": "7.7.1",
      "type": "metricbeat",
      "ephemeral_id": "1e80477c-75e3-4001-b80a-88421bc48b3f",
      "hostname": "d0ab35b86d7a",
      "id": "d8079cbc-5971-46ce-b8d5-c18500d6b484"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-06-09T01:30:12.793Z"
    ]
  },
  "sort": [
    1591666212793
  ]
}

For reference, this is the output from the Prometheus exporter when I make a GET request to it.

# HELP method_call_total The number of times a method is called
# TYPE method_call_total counter
method_call_total{method_name="run"} 227.0
# HELP method_call_created The number of times a method is called
# TYPE method_call_created gauge
method_call_created{method_name="run"} 1.5916654136987088e+09
# HELP method_duration The duration of the method
# TYPE method_duration histogram
method_duration_bucket{le="0.005",method_name="run"} 225.0
method_duration_bucket{le="0.01",method_name="run"} 226.0
method_duration_bucket{le="0.025",method_name="run"} 227.0
method_duration_bucket{le="0.05",method_name="run"} 227.0
method_duration_bucket{le="0.075",method_name="run"} 227.0
method_duration_bucket{le="0.1",method_name="run"} 227.0
method_duration_bucket{le="0.25",method_name="run"} 227.0
method_duration_bucket{le="0.5",method_name="run"} 227.0
method_duration_bucket{le="0.75",method_name="run"} 227.0
method_duration_bucket{le="1.0",method_name="run"} 227.0
method_duration_bucket{le="2.5",method_name="run"} 227.0
method_duration_bucket{le="5.0",method_name="run"} 227.0
method_duration_bucket{le="7.5",method_name="run"} 227.0
method_duration_bucket{le="10.0",method_name="run"} 227.0
method_duration_bucket{le="+Inf",method_name="run"} 227.0
method_duration_count{method_name="run"} 227.0
method_duration_sum{method_name="run"} 0.13504729981650598

(The counts don't line up as they were two separate metric requests).

The application is a Python application instrumented with the Python Prometheus client, https://github.com/prometheus/client_python

This is what the metricbeat.yml configuration that I am using looks like:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

# Metrics collected from a Prometheus endpoint
metricbeat.modules:
  - module: prometheus
    period: 5s
    hosts: ["prometheus-test-service:8081"]
    metrics_path: /prometheus/metrics
    use_types: true
    rate_counters: true

Here is the docker-compose.yaml file I am using to run the ELK stack:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.1
    container_name: elasticsearch
    ports:
        - 9200:9200
    environment:
        - discovery.type=single-node
        - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        - "cluster.routing.allocation.disk.threshold_enabled=false"
        - "http.cors.enabled=true"
        - "http.cors.allow-origin=*"
        - "http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization"
        - "http.cors.allow-credentials=true"
        - "xpack.security.enabled=false"
    healthcheck:
      test: curl -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  kibana:
    image: docker.elastic.co/kibana/kibana:7.7.1
    container_name: kibana
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    healthcheck:
      test: curl -s https://localhost:5601 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  metricbeat:
    image: docker.elastic.co/beats/metricbeat:7.7.1
    container_name: metricbeat
    volumes:
      # Mount the metricbeat config file that includes the prometheus exporter.
      - ./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
    command: --strict.perms=false -e  # -e flag to log to stderr and disable syslog/file output
    depends_on:
        - elasticsearch
        - kibana
    healthcheck:
      test: metricbeat test config
      interval: 30s
      timeout: 15s
      retries: 5

When I boot up Metricbeat, I do see these two logs:

metricbeat               | 2020-06-09T01:50:00.671Z	WARN	[cfgwarn]	collector/data.go:47	BETA: Prometheus 'use_types' setting is beta
metricbeat               | 2020-06-09T01:50:00.671Z	WARN	[cfgwarn]	collector/data.go:50	EXPERIMENTAL: Prometheus 'rate_counters' setting is experimental

Any help would be appreciated. Thank you!

2

Hi @Jose_Vargas!

It looks to me that all the metrics are treated like gauges and not counters. This means that Metricbeat understands all the metrics as gauges and handles them like this.

From what I see method_call_total is reported as counter in the exported output you shared, but it seems that Metricbeat is not detecting it.

Could you please verify that your exporter serves the metrics correctly?

C.

cc: @exekias

3

Thanks for the quick response @ChrsMark! Here is what I did to verify my exporter.

I added a separate Prometheus container to scrape my exporter and I queried the target metadata endpoint of the Prometheus process, HTTP API | Prometheus, for the method_call metric. Here is the response:

$ curl -G http://localhost:9090/api/v1/targets/metadata --data-urlencode 'metric=method_call' | jq '.'
{
  "status": "success",
  "data": [
    {
      "target": {
        "instance": "prometheus-test-service:8081",
        "job": "prometheus"
      },
      "type": "counter",
      "help": "The number of times a method is called",
      "unit": ""
    }
  ]
}

so Prometheus itself seems to identify the metric as a counter.

Here are the response headers from the exporter endpoint:

$ curl -v localhost:8081/prometheus/metrics
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET /prometheus/metrics HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Server: TwistedWeb/18.4.0
< Date: Tue, 09 Jun 2020 14:07:09 GMT
< Content-Type: text/plain; version=0.0.4; charset=utf-8

Please let me know if there is something else that I can do to to verify the Prometheus exporter.

Thanks!

4

Hi!

I found the time to try this one out and I used your prometheus metrics as an input for the module's unit tests. The generated output is:

   {
        "event": {
            "dataset": "prometheus.collector",
            "duration": 115000,
            "module": "prometheus"
        },
        "metricset": {
            "name": "collector",
            "period": 10000
        },
        "prometheus": {
            "labels": {
                "instance": "127.0.0.1:49956",
                "job": "prometheus",
                "method_name": "run"
            },
            "method_call_created": {
                "value": 1591665413.6987088
            },
            "method_call_total": {
                "counter": 227,
                "rate": 0
            },
            "method_duration": {
                "histogram": {
                    "counts": [
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0
                    ],
                    "values": [
                        0.0025,
                        0.0075,
                        0.0175,
                        0.037500000000000006,
                        0.0625,
                        0.0875,
                        0.175,
                        0.375,
                        0.625,
                        0.875,
                        1.75,
                        3.75,
                        6.25,
                        8.75,
                        12.5
                    ]
                }
            }
        },
        "service": {
            "address": "127.0.0.1:55555",
            "type": "prometheus"
        }
    }

This means that I cannot reproduce it. Feel free to provide me any other content I could use to reproduce your case.

Something important here is that rates are not supported on oss version of metricbeat. What version do you use and how you install it?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.