Failed to get docker stats: request returned Not Found for API route and version

Elastic Stack Beats
,
1

I am using metricbeat in conjunction with Docker to get metrics from a given container via hints autodiscovery.

I don't think its working properly as I am getting an error, in the error message key:

failed to get docker stats: request returned Not Found for API route and version http://172.20.0.2:8101/v1.24/containers/json?limit=0, check if the server supports the requested API version

It appears to be targeting the correct container's IP address upon inspection of the json.

Here is my metricbeat and demo app in docker compose:

 metricbeat:
    container_name: metricbeat
    user: root
    image: mymetricbeat:docker
    volumes:
    # needed to access additional information about containers
    - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
    - esnet
  myloggingapp:
    container_name: loggingapp
    image: loggingapp:latest
    environment: 
      - ASPNETCORE_URLS=http://+:8101
    labels:
      co.elastic.metrics/enabled: true
      co.elastic.metrics/module: docker
      co.elastic.metrics/metricsets: cpu
      co.elastic.metrics/hosts: '$${data.host}:8101'
      co.elastic.metrics/period: 1m
    networks:
    - esnet

Here is the yml:

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true                

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "sample2-%{+YYYY.MM.dd}"

setup.template.name: "sample2"
setup.template.pattern: "sample2-*"

Also, for the event key, is the CPU statistic correct as it states duration : 2090999 as I'm struggling to make sense of that.

2

Hey @daz1761,

The docker module is intended to be used in the host, to collect information from the daemon about all the containers and other resources there, it should be configured in your metricbeat container, not in the hints of specific containers.

Something like this:

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true       

metricbeat.modules:
  - name: docker
    metricsets:
      - cpu
      ...
    hosts: ["unix:///var/run/docker.sock"]

output.elasticsearch:
...

You can find a reference configuration here: https://github.com/elastic/beats/blob/v7.6.2/deploy/docker/metricbeat.docker.yml

Sorry, I think I don't follow this, what event key do you mean? and where does this duration appear?

3

Thanks for the reply, just to move things forward, I went with this configuration which gets the stats from every single container running:

    metricbeat.modules:
module: docker
          metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
          hosts: ["unix:///var/run/docker.sock"]
          period: 1m
          enabled: true

So if I want to use hints to only retrieve information about specific container i.e. my logging app, are you saying I need this configuration:

 metricbeat.modules:
module: docker
          metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
          hosts: ["unix:///var/run/docker.sock"]
          period: 1m
          enabled: true

AND

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

And in my docker compose on my logging app:

myloggingapp:
    container_name: loggingapp
    image: loggingapp:latest
    environment: 
      - ASPNETCORE_URLS=http://+:8101
    labels:
      co.elastic.metrics/enabled: true
      co.elastic.metrics/module: docker
      co.elastic.metrics/metricsets: cpu
      co.elastic.metrics/hosts: '$${data.host}:8101'
      co.elastic.metrics/period: 1m

Is this what is needed to get the statistics of a single container? Or do I just need this as a label?

co.elastic.metrics/enabled: true

Thanks

4

The docker module always collects metrics from all the containers. There is no way of getting metrics from a single container. Something you can do is to add a processor that drops all events not coming from an specific container.

For example, something like this would discard all events not coming from the loggingapp container:

metricbeat.modules:
  - name: docker
    metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
    period: 1m
    hosts: ["unix:///var/run/docker.sock"]
    processors:
      - drop_event:
          when:
            not.equals.docker.container.name: loggingapp

Hints in labels (like co.elastic.metrics/module) can be used to monitor specific applications, for example if your application were exposing prometheus metrics, you could get them with a configuration like the following one:

myloggingapp:
  ...
  labels:
    co.elastic.metrics/enabled: true
    co.elastic.metrics/module: prometheus
    co.elastic.metrics/metricsets: collector
    co.elastic.metrics/hosts: '$${data.host}:9090'
    co.elastic.metrics/period: 1m
5

Hi @jsoriano,

Thanks again for the reply...

I think I understand the hints thing now in terms of why it didn't work correctly when trying to run the Docker module directly on a container, and how the hints are more designed to be used with other modules such as nginx, etc. that takes specific application metrics from a container, which I will be exploring at some point.

Interestingly after your processor example, it reminded me of another solution I tried to carry out that used a condition on the label, but it didn't work:

metricbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains
                docker.container.labels: ["jupiter.metrics/enabled: true"]
            config:
            - module: docker
              metricsets: ["cpu", "info"]
              period: 1m
              hosts: ["unix:///var/run/docker.sock"]
              enabled: true

Then in my docker-compose:

myloggingapp:
    container_name: loggingapp
    image: loggingapp:latest
    #command: -f JSON -m 10
    environment: 
      - ASPNETCORE_URLS=http://+:8101
    labels:
      jupiter.metrics/enabled: true

Can something like this be done?

Thank you.

6

The problem with this configuration is similar to the previous cases, the docker module collects metrics from all the running containers. If you use this template, it will start a docker module for each container with these labels, leading to duplicated metrics of all running containers. The docker module is intended to be used in the main configuration, not per container in an autodiscover configuration.

You can though take advantage of this label if you want to discard all the metrics from containers that don't contain it, following with my example, with a condition like this one:

metricbeat.modules:
  - name: docker
    metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
    period: 1m
    hosts: ["unix:///var/run/docker.sock"]
    processors:
      - drop_event:
          when:
            not.equals.docker.container.labels:
              "jupiter_metrics/enabled": "true"
7

Thanks again @jsoriano

That makes sense as you certainly don't want multiple metricbeat containers running for each container, as eventually there will be a lot of containers/microservices running in our system.

I will try this tomorrow to see if it works then we have the 2 options to work with - either collect metrics for every container, or take advantage of the custom label in conjunction with a condition.

Will let you know asap how I get on.

Thanks again.

8

Hi @jsoriano,

I've just tried your conditional solution above and it didn't work. Is there something I am missing?

metricbeat.modules:
- module: docker
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  period: 1m
  #enabled: true
  processors:
    - drop_event:
        when:
          not.equals.docker.container.labels:
            "jupiter_metrics/enabled": "true"

// docker compose
myloggingapp:
    container_name: loggingapp
    image: loggingapp:docker
    environment: 
      - ASPNETCORE_URLS=http://+:8101
    labels:
      # custom conditional label
      jupiter_metrics/enabled: true

I have tried escaping the / in the label and adding enabled: true and removing the quotes from the boolean true in the label with your solution, but this does not work either. Its not picking up any stats and thus not creating an index in elasticsearch.

I've checked the label is present in the logging app also. To note the metricbeat.modules is the only thing I have in the config apart from the elasticsearch output. Does hints still need to be present when working with this solution?

Thanks

9

Hi @jsoriano,

Did you receive my last few posts?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.