Cannot get Autodiscover conditions to work

Elastic Stack Beats
,
1

I have a situation where I am trying to get filebeat-7.16.2 to ship logs from three separate docker containers on a single host.

I am trying to use autodiscover to launch different configurations for the logs from each container, but cannot get it to work. For some reason each log from any container gets sent with every configuration. So each log line gets parsed three times, and imported erroneously twice to ealsticsearch.

I assume something is going wrong with the conditions that I am giving the autodiscover templates.

My configuration is as follows:

filebeat.autodiscover.providers:
  - type: docker
    templates:
      - condition:
        contains:
          container.image.name: frontend-image
        config:
          - module: nginx
            access:
              input:
                type: container
                paths:
                  - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
                fields_under_root: true
                fields:
                  service.name: frontend-nginx
                  service.environment: staging
                stream: stdout
            error:
              enabled: true
              input:
                type: container
                paths:
                  - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
                fields_under_root: true
                fields:
                  service.name: frontend
                  service.environment: staging
                stream: stderr
      - condition:
        contains:
          container.image.name: backend-image
        config:
          - type: container
            paths:
              - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
            json.keys_under_root: true
            json.overwrite_keys: true
            json.add_error_key: true
            json.expand_keys: true
            pipeline: custom-pipeline
            fields_under_root: true
            fields:
              service.name: backend
              service.environment: staging
      - condition:
        contains:
          container.image.name: auth-image
        config:
          - type: container
            paths:
              - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
            json.keys_under_root: true
            json.overwrite_keys: true
            json.add_error_key: true
            json.expand_keys: true
            pipeline: custom-pipeline
            fields_under_root: true
            fields:
              service.name: auth
              service.environment: staging

Some things to note on the above, my conditions are currently set to container.image.name. The autodiscover docs suggest docker.container.image or docker.container.name. I have tried both of these with the same result.

I attempted container.image.name as when inspecting the event in the filebeat logs, it contains data in docker.container.labels, but no docker.container.name property.

It does have a container attribute that contains name, image and id etc.

Am I correct in assuming that the conditions refer to the data in the emitted event? So under the hood, filebeat monitors the output of all the containers, and subsquently uses the filter to pick a configuration to use with the event?

Or does it launch a separate watcher per configuration? I.E the condition does not refer to the event, but to some other data, and then filebeat uses this to fill in the data in the data.docker.container.id section?

Also, what's the default behavior when no conditions are met? Does it default to the first config, or just not send the event at all?

Any help would be much appreciated, thanks in advance...

2

Found the issue above. YAML whitespace issue.

- condition:
  contains:
    docker.container.image: frontend

Should actually be:

- condition:
    contains:
      docker.container.image: frontend

:exploding_head:

1 Like