Collecting logs using docker.container.labels condition with filebeat:8.5.3

[Viktor_Herman]

I'm having some problems configuring filebeat to only ingest the logs from the containers that I want. I exhausted all of the resources and documentation doesn't have any examples on this exact issue. Filebeat version is 8.5.3. It is still the default image taken from dockerhub, but pushed to my registry. I'm using this with Docker Swarm and need to ship my logs from containers labeled with "filebeat.enable=enabled" to my ES instance. Logstash takes the input from filebeat and sends it to ES.

Filebeat stack file:

version: '3.8'

services:
  filebeat:
    image: my.registry.something/filebeat:8.5.3      
    user: root
    command:
      - -e
      - --strict.perms=false    
    volumes:
      - ./extensions/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro,Z
      - /var/log:/var/log:ro,Z
      - /var/lib/docker/containers:/var/lib/docker/containers:ro,Z
      - /var/run/docker.sock:/var/run/docker.sock:ro,Z
      - filebeatdata:/usr/share/filebeat/data
    deploy:
      labels:
        - "filebeat.enable=enabled"
      mode: global    
    env_file:
      - ./.envext      
    networks:
      - elk
    depends_on:
      - elasticsearch
      - logstash
      - kibana  
networks:
  elk:
    external: true
volumes:
  filebeatdata:
    driver: glusterfs
    name: "gv0/filebeatdata"

filebeat.yml configuration file:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config.enabled: false
      templates:
        - condition:
            equals.docker.container.labels.filebeat.enable: "enabled"
          config:
            - type: container
              paths:
                - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
              fields:
                event.dataset: "${data.docker.container.image}"
              fields_under_root: true

monitoring:
  enabled: true
  elasticsearch:
    hosts: [ "http://elasticsearch:9200" ]       
    username: beats_system
    password: ${BEATS_SYSTEM_PASSWORD}

output.logstash:
  enabled: true
  hosts: [ "logstash:5044" ]
  username: elastic
  password: ${ELASTIC_PASSWORD}

setup.kibana:
  host: http://kibana:5601
  username: elastic
  password: ${ELASTIC_PASSWORD}

setup.dashboards.enabled: true

http:
  enabled: true
  host: 0.0.0.0

In theory all of this looks right but I'm not getting any output. There is something wrong with my condition, because if I remove it I get the logs in the right format, but I get ALL of the logs - from all of the containers, not just the ones I want. In this case I should get only logs from filebeat itself, with event.dataset set to ${data.container.image}, which would be my.registry.something/filebeat:8.5.3

Elastic is awesome and beats are the way to go when adding functionality to your Elastic stack, but dear God, please allow me to add some things to your documentation when all of this is over. I'll do it for free, so no-one has to go through all of this again.

[Viktor_Herman]

Hey guys, today is your lucky day!
My configuration was right all along, but I found out something about Docker itself.

If you take a look at my stack file label was deployed as:

deploy:
   labels:
      - "filebeat.enable=enabled"

This actually attaches the label to service, instead of the container.
So the right way to do this is:

labels:
   - "filebeat.enable=enabled"

I actually changed "enabled" to "true" and "filebeat.enable" to "filebeat_enable"

So in the end, all of this looks something like this:
filebeat-stack.yml:

version: '3.8'

services:
  filebeat:
    image: my.registry/filebeat:8.5.3      
    user: root
    command:
      - -e
      - --strict.perms=false    
    volumes:
      - ./extensions/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro,Z
      - /var/log:/var/log:ro,Z
      - /var/lib/docker/containers:/var/lib/docker/containers:ro,Z
      - /var/run/docker.sock:/var/run/docker.sock:ro,Z
      - filebeatdata:/usr/share/filebeat/data
    deploy:
      mode: global 
    labels:
      - "filebeat_enable=true"      
    env_file:
      - ./.envext      
    networks:
      - elk
    depends_on:
      - elasticsearch
      - logstash
      - kibana  
networks:
  elk:
    external: true
volumes:
  filebeatdata:
    driver: glusterfs
    name: "gv0/filebeatdata"

filebeat.yml config file:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals:
              docker.container.labels.filebeat_enable: "true"      
          config:
            - type: container
              containers.ids:
                - "${data.docker.container.id}"
              paths:
                - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
              fields:
                event.dataset: "${data.docker.container.image}"
              fields_under_root: true

processors:
  - add_docker_metadata: ~      
              

monitoring:
  enabled: true
  elasticsearch:
    hosts: [ "http://elasticsearch:9200" ]       
    username: beats_system
    password: ${BEATS_SYSTEM_PASSWORD}

output.logstash:
  enabled: true
  hosts: [ "logstash:5044" ]
  username: elastic
  password: ${ELASTIC_PASSWORD}

setup.kibana:
  host: http://kibana:5601
  username: elastic
  password: ${ELASTIC_PASSWORD}

setup.dashboards.enabled: true

http:
  enabled: true
  host: 0.0.0.0

This can be referenced here: Compose file version 3 reference | Docker Documentation

In the end, filebeat works as it should, but Docker labels get attached to different things if used in those two formats. I hope this saves you some time.