AutoDiscover - can't ship docker container logs by image name

Elastic Stack Beats
1

I'm trying to set up the AutoDiscover component in the Filebeat.yml.
I'm using it in order to take logs from specific containers which generated by a specific Docker image.
When I'm working with the "filebeat.autodiscover" should I need to add the "filebeat.prospectors" block too?
Currently, this is my configuration without filebeat.prospectors:

filebeat.autodiscover:
# List of enabled autodiscover providers
providers:
- type: docker
templates:
- condition:
equals.docker.container.image: wildfly_image
config:
- type: log
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
processors:
- add_docker_metadata: ~

fields:
environment: WF-Prod
fields_under_root: true

multiline.pattern: '^[[:space:]]+|]$'
multiline.match: after

As you can see, because of the "filebeat.prospectors" has been omitted, I had to add the processors, fields & multiline options under the "filebeat.autodiscover". Is it the right thing to do?

2

Your setup looks good to me. I guess either a problem with your config (indentation?) or the docker container image is wrong.

Can you share your whole filebeat.yml file and the debug log of filebeat run?

3

The issue was the indentation, and "add_docker_metadata" should be omitted.
Working configuration is below:

filebeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: docker
      templates:
        - condition:
            equals.docker.container.image: full_image_path/image_name:version
          config:
            - type: log
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              fields:
                environment: Development
              fields_under_root: true

              multiline.pattern: '^[[:space:]]+|]$'
              multiline.match: after

Thanks a lot!

1 Like
4

I'm glad you discovered the original issue @Asaf_Shabat,

Also you may want to use the new docker prospector, available since 6.1, prospector conf would look like this:

type: docker
containers.ids: ${data.docker.container.id}
multiline.pattern: '^[[:space:]]+|]$'
multiline.match: after
5

Great, but how could I filter the containers by image name without the using of autodiscover?

6

Not at the moment, docker prospector is dumb, it only accepts container ids as it's what it uses to search for the logs. Autodiscover is in place to do what you want.

Best regards

7

Also I forgot to mention, you are using equals condition, but you have all existing conditions available, just in case you want to match only some part of the image, for instance.

8

Thank you very much!
It helps!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.