AutoDiscover - can't ship docker container logs by image name


(Asaf Shabat) #1

I'm trying to set up the AutoDiscover component in the Filebeat.yml.
I'm using it in order to take logs from specific containers which generated by a specific Docker image.
When I'm working with the "filebeat.autodiscover" should I need to add the "filebeat.prospectors" block too?
Currently, this is my configuration without filebeat.prospectors:

filebeat.autodiscover:
# List of enabled autodiscover providers
providers:
- type: docker
templates:
- condition:
equals.docker.container.image: wildfly_image
config:
- type: log
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
processors:
- add_docker_metadata: ~

fields:
environment: WF-Prod
fields_under_root: true

multiline.pattern: '^[[:space:]]+|]$'
multiline.match: after

As you can see, because of the "filebeat.prospectors" has been omitted, I had to add the processors, fields & multiline options under the "filebeat.autodiscover". Is it the right thing to do?


(Adrian Serrano) #2

Your setup looks good to me. I guess either a problem with your config (indentation?) or the docker container image is wrong.

Can you share your whole filebeat.yml file and the debug log of filebeat run?


(Asaf Shabat) #3

The issue was the indentation, and "add_docker_metadata" should be omitted.
Working configuration is below:

filebeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: docker
      templates:
        - condition:
            equals.docker.container.image: full_image_path/image_name:version
          config:
            - type: log
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              fields:
                environment: Development
              fields_under_root: true

              multiline.pattern: '^[[:space:]]+|]$'
              multiline.match: after

Thanks a lot!


(Carlos Pérez Aradros) #4

I'm glad you discovered the original issue @Asaf_Shabat,

Also you may want to use the new docker prospector, available since 6.1, prospector conf would look like this:

type: docker
containers.ids: ${data.docker.container.id}
multiline.pattern: '^[[:space:]]+|]$'
multiline.match: after

(Asaf Shabat) #5

Great, but how could I filter the containers by image name without the using of autodiscover?


(Carlos Pérez Aradros) #6

Not at the moment, docker prospector is dumb, it only accepts container ids as it's what it uses to search for the logs. Autodiscover is in place to do what you want.

Best regards


(Carlos Pérez Aradros) #7

Also I forgot to mention, you are using equals condition, but you have all existing conditions available, just in case you want to match only some part of the image, for instance.


(Asaf Shabat) #8

Thank you very much!
It helps!


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.