# Failed to get docker stats: request returned Not Found for API route and version

**URL:** <https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720>\
**Category:** Beats\
**Tags:** docker, metricbeat\
**Created:** [May 1, 2020, 2:25pm UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720 "2020-05-01T14:25:11Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![daz1761](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daz1761/32/61856_2.png) [@daz1761](https://discuss.elastic.co/u/daz1761)\
**Post date:** [May 1, 2020, 2:25pm UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/1 "2020-05-01T14:25:11Z")

</div>

I am using metricbeat in conjunction with Docker to get metrics from a given container via hints autodiscovery.

I don't think its working properly as I am getting an error, in the error message key:

failed to get docker stats: request returned Not Found for API route and version [http://172.20.0.2:8101/v1.24/containers/json?limit=0](http://172.20.0.2:8101/v1.24/containers/json?limit=0), check if the server supports the requested API version

It appears to be targeting the correct container's IP address upon inspection of the json.

Here is my metricbeat and demo app in docker compose:

```
 metricbeat:
    container_name: metricbeat
    user: root
    image: mymetricbeat:docker
    volumes:
    # needed to access additional information about containers
    - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
    - esnet
  myloggingapp:
    container_name: loggingapp
    image: loggingapp:latest
    environment: 
      - ASPNETCORE_URLS=http://+:8101
    labels:
      co.elastic.metrics/enabled: true
      co.elastic.metrics/module: docker
      co.elastic.metrics/metricsets: cpu
      co.elastic.metrics/hosts: '$${data.host}:8101'
      co.elastic.metrics/period: 1m
    networks:
    - esnet 

```

Here is the yml:

```
metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true                

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "sample2-%{+YYYY.MM.dd}"

setup.template.name: "sample2"
setup.template.pattern: "sample2-*"

```

Also, for the event key, is the CPU statistic correct as it states duration : 2090999 as I'm struggling to make sense of that.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 11, 2020, 2:45pm UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/2 "2020-05-11T14:45:18Z")

</div>

Hey @daz1761,

The docker module is intended to be used in the host, to collect information from the daemon about all the containers and other resources there, it should be configured in your metricbeat container, not in the hints of specific containers.

Something like this:

```auto
metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true       

metricbeat.modules:
  - name: docker
    metricsets:
      - cpu
      ...
    hosts: ["unix:///var/run/docker.sock"]

output.elasticsearch:
...

```

You can find a reference configuration here: [https://github.com/elastic/beats/blob/v7.6.2/deploy/docker/metricbeat.docker.yml](https://github.com/elastic/beats/blob/v7.6.2/deploy/docker/metricbeat.docker.yml)

> [@daz1761](#):
>
> Also, for the event key, is the CPU statistic correct as it states duration : 2090999 as I'm struggling to make sense of that.

Sorry, I think I don't follow this, what event key do you mean? and where does this duration appear?

---

<div class="post-metadata">

**Author:** ![daz1761](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daz1761/32/61856_2.png) [@daz1761](https://discuss.elastic.co/u/daz1761)\
**Post date:** [May 11, 2020, 3:51pm UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/3 "2020-05-11T15:51:43Z")

</div>

Thanks for the reply, just to move things forward, I went with this configuration which gets the stats from every single container running:

```
    metricbeat.modules:
module: docker
          metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
          hosts: ["unix:///var/run/docker.sock"]
          period: 1m
          enabled: true

```

So if I want to use hints to only retrieve information about specific container i.e. my logging app, are you saying I need this configuration:

```
 metricbeat.modules:
module: docker
          metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
          hosts: ["unix:///var/run/docker.sock"]
          period: 1m
          enabled: true

```

AND

```
metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

```

And in my docker compose on my logging app:

```
myloggingapp:
    container_name: loggingapp
    image: loggingapp:latest
    environment: 
      - ASPNETCORE_URLS=http://+:8101
    labels:
      co.elastic.metrics/enabled: true
      co.elastic.metrics/module: docker
      co.elastic.metrics/metricsets: cpu
      co.elastic.metrics/hosts: '$${data.host}:8101'
      co.elastic.metrics/period: 1m

```

Is this what is needed to get the statistics of a single container? Or do I just need this as a label?

> co.elastic.metrics/enabled: true

Thanks

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 11, 2020, 7:10pm UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/4 "2020-05-11T19:10:27Z")

</div>

The docker module always collects metrics from all the containers. There is no way of getting metrics from a single container. Something you can do is to add a processor that drops all events not coming from an specific container.

For example, something like this would discard all events not coming from the loggingapp container:

```auto
metricbeat.modules:
  - name: docker
    metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
    period: 1m
    hosts: ["unix:///var/run/docker.sock"]
    processors:
      - drop_event:
          when:
            not.equals.docker.container.name: loggingapp

```

Hints in labels (like `co.elastic.metrics/module`) can be used to monitor specific applications, for example if your application were exposing prometheus metrics, you could get them with a configuration like the following one:

```auto
myloggingapp:
  ...
  labels:
    co.elastic.metrics/enabled: true
    co.elastic.metrics/module: prometheus
    co.elastic.metrics/metricsets: collector
    co.elastic.metrics/hosts: '$${data.host}:9090'
    co.elastic.metrics/period: 1m

```

---

<div class="post-metadata">

**Author:** ![daz1761](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daz1761/32/61856_2.png) [@daz1761](https://discuss.elastic.co/u/daz1761)\
**Post date:** [May 11, 2020, 8:39pm UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/5 "2020-05-11T20:39:28Z")

</div>

Hi @jsoriano,

Thanks again for the reply...

I think I understand the hints thing now in terms of why it didn't work correctly when trying to run the Docker module directly on a container, and how the hints are more designed to be used with other modules such as nginx, etc. that takes specific application metrics from a container, which I will be exploring at some point.

Interestingly after your processor example, it reminded me of another solution I tried to carry out that used a condition on the label, but it didn't work:

```auto
metricbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains
                docker.container.labels: ["jupiter.metrics/enabled: true"]
            config:
            - module: docker
              metricsets: ["cpu", "info"]
              period: 1m
              hosts: ["unix:///var/run/docker.sock"]
              enabled: true

```

Then in my docker-compose:

```auto
myloggingapp:
    container_name: loggingapp
    image: loggingapp:latest
    #command: -f JSON -m 10
    environment: 
      - ASPNETCORE_URLS=http://+:8101
    labels:
      jupiter.metrics/enabled: true

```

Can something like this be done?

Thank you.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 12, 2020, 10:34am UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/6 "2020-05-12T10:34:15Z")

</div>

The problem with this configuration is similar to the previous cases, the docker module collects metrics from all the running containers. If you use this template, it will start a docker module for each container with these labels, leading to duplicated metrics of all running containers. The docker module is intended to be used in the main configuration, not per container in an autodiscover configuration.

You can though take advantage of this label if you want to discard all the metrics from containers that don't contain it, following with my example, with a condition like this one:

```auto
metricbeat.modules:
  - name: docker
    metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
    period: 1m
    hosts: ["unix:///var/run/docker.sock"]
    processors:
      - drop_event:
          when:
            not.equals.docker.container.labels:
              "jupiter_metrics/enabled": "true"

```

---

<div class="post-metadata">

**Author:** ![daz1761](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daz1761/32/61856_2.png) [@daz1761](https://discuss.elastic.co/u/daz1761)\
**Post date:** [May 12, 2020, 6:25pm UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/7 "2020-05-12T18:25:33Z")

</div>

Thanks again @jsoriano

That makes sense as you certainly don't want multiple metricbeat containers running for each container, as eventually there will be a lot of containers/microservices running in our system.

I will try this tomorrow to see if it works then we have the 2 options to work with - either collect metrics for every container, or take advantage of the custom label in conjunction with a condition.

Will let you know asap how I get on.

Thanks again.

---

<div class="post-metadata">

**Author:** ![daz1761](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daz1761/32/61856_2.png) [@daz1761](https://discuss.elastic.co/u/daz1761)\
**Post date:** [May 13, 2020, 9:39am UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/8 "2020-05-13T09:39:14Z")

</div>

Hi @jsoriano,

I've just tried your conditional solution above and it didn't work. Is there something I am missing?

```
metricbeat.modules:
- module: docker
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  period: 1m
  #enabled: true
  processors:
    - drop_event:
        when:
          not.equals.docker.container.labels:
            "jupiter_metrics/enabled": "true"

// docker compose
myloggingapp:
    container_name: loggingapp
    image: loggingapp:docker
    environment: 
      - ASPNETCORE_URLS=http://+:8101
    labels:
      # custom conditional label
      jupiter_metrics/enabled: true

```

I have tried escaping the `/` in the label and adding `enabled: true` and removing the quotes from the boolean true in the label with your solution, but this does not work either. Its not picking up any stats and thus not creating an index in elasticsearch.

I've checked the label is present in the logging app also. To note the `metricbeat.modules` is the only thing I have in the config apart from the elasticsearch output. Does hints still need to be present when working with this solution?

Thanks

---

<div class="post-metadata">

**Author:** ![daz1761](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daz1761/32/61856_2.png) [@daz1761](https://discuss.elastic.co/u/daz1761)\
**Post date:** [May 18, 2020, 3:52pm UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/9 "2020-05-18T15:52:06Z")

</div>

Hi @jsoriano,

Did you receive my last few posts?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 15, 2020, 3:52pm UTC](https://discuss.elastic.co/t/failed-to-get-docker-stats-request-returned-not-found-for-api-route-and-version/230720/10 "2020-06-15T15:52:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.