Filebeat Autodiscovery: Filtering by container labels

[ppik (Peeter Piksarv)]

Hi,
I would like to set up Filebeat configuration with docker autodiscovery provider to create prospectors only for docker containers with certain label, e.g., filebeat.enable: "true". However I find it difficult to find the correct condition format to achieve this.

I have tried the following config, but it does not seem to match any docker events:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals:
              docker.container.labels.filebeat.enable: "true"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"

That is, I see the following in the filebeat log:

2018-02-16T14:24:26.330Z        DEBUG   [docker]        docker/watcher.go:207   Got a new docker event: {create 93d527f6de152c3735a19517a84f444644cdd1d2fb16c11aaae202d40ffea8b3 local/hello:dev container create {93d527f6de152c3735a19517a84f444644cdd1d2fb16c11aaae202d40ffea8b3 map[com.docker.compose.version:1.18.0 image:local/hello:dev maintainer:NGINX Docker Maintainers <docker-maint@nginx.com> name:api_hello_1 com.docker.compose.container-number:1 com.docker.compose.project:api com.docker.compose.service:hello filebeat.enable:true com.docker.compose.config-hash:57c6c7591874f57f02a64297c36cc71e96085fa3c690170bd7cd75e9a42a34ad com.docker.compose.oneoff:False]} local 1518791066 1518791066330127148}

But it does not trigger autodiscover start event as can be seen with filtering based on docker.container.name as in the documentation example.

Could anyone help me out with the correct condition format for docker.container.labels?

[exekias (Carlos Pérez Aradros)]

Hi @ppik,

Could you try to use a label without dots on it? I think there is an issue with labels containing dots at the moment. We saw something similar in the Kubernetes provider and fixed it.

Best regards

[ppik (Peeter Piksarv)]

Hi @exekias,

Thank you for the suggestion. Unfortunately it doesn't help if I remove the dot from the filebeat config above, the started docker containers with the right labels seem to be left undiscovered. Tried also contains and regexp matchers, but with no luck.

[exekias (Carlos Pérez Aradros)]

Sorry, I meant removing the dot both from the label and the config. Dots are not supported as of today, we are working on a fix for this

[ppik (Peeter Piksarv)]

Thanks for the clarification. For the minimal configuration I have something like this, currently:
filebeat.yml

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.labels.somelabel: "somevalue"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"

output.console:
  pretty: true

and i start it with the following docker-compose.yml:

version: '3'

services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:6.2.1
    command: "--strict.perms=false -v -e -d autodiscover,docker"
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /var/lib/docker/containers:/var/lib/docker/containers
      - /var/run/docker.sock:/var/run/docker.sock

  test:
    image: alpine
    command: "sh -c 'while true; do echo test; sleep 1; done'"
    depends_on:
      - filebeat
    labels:
      somelabel: "somevalue"

Docker events can be seen by the filebeat, but the autodiscovery seems to miss the condition.

[exekias (Carlos Pérez Aradros)]

Hi again,

Thank you for the testing files, they helped me reproducing this issue.

I'm afraid you found a bug, the good news is that I have a working fix that should go in soon, here you have the code change so you can follow it's release or build your own image: https://github.com/elastic/beats/pull/6412

Thank you!

Best regards