Filebeat autodiscover for docker does not work on /var/lib/docker/containers


(Roger L.) #1

Hi,

I am quite puzzled about the autodiscover feature for "tea"ing docker logs.
This look quite useful, but despite reading the documentation and the few posts about it, I could not manage to have it fully work.
To be clear : I managed to get a copy of the logs from within /var/lib/docker/volumes/whatever, but I did not manage to perform the same with /var/lib/docker/containers

Filebeat runs as long as the elk stack on a swarm environnement.
They are running under their own network, thus are identified by the service name (e.g. logstash:5044)

Here's my config ::

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        # conditions are not compulsory. I have tried to remove them, but the problem remains
        - condition.or:
          - contains.docker.container.name: "apache"
          - contains.docker.container.name: "nginx"
          config:
          - type: docker
            containers.ids:
               - ${data.docker.container.id}
            multiline:
                pattern: '^\[#|\d{4}'
                negate:  true
                match:   after
          - type: log
            paths: # various attempts
            - /var/lib/docker/containers/${data.docker.container.id}/*.log # KO
            - /var/lib/docker/containers/631c570fcd974c248142726cf9c4a1aa21d45cc7b9a39a0eed8dc73eeb41d1df/*.log #KO
            - /var/lib/docker/containers/*/*.log #KO
            - /var/lib/docker/volumes/${data.docker.container.name}/_data/localhost_access_log*.txt #OK
            - /var/lib/docker/volumes/whoAmTest_tmpvol*/_data/*.log #OK

# I tried to add this later, according to a post I read here : https://discuss.elastic.co/t/problem-getting-autodiscover-docker-to-work-with-filebeat/144349/10
filebeat.inputs:
  - type: docker
    containers.ids:
      - "*"
			
processors:
 - add_docker_metadata:
     host: "unix:///var/run/docker.sock"

output.logstash:
  hosts: ["logstash:5044"]
  bulk_max_size: 4096

I know some paths are not following the documentation guidelines, but there are here for the sole purpose of giving a few clues about the problem (I hope!).

I see no error on filebeat logs, nor on ES or logstash.
It's just not working for the logs from /var/lib/docker/containers//.log!

Any idea / hint?


(Archimedes Trajano) #2

I have a similar issue, but I am new to setting this all up. Here's my filebeats.yml which I bake into the image

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
  - drop_fields:
      fields:
        - "docker.containers.labels"
  - add_host_metadata: ~
  - add_cloud_metadata: ~

output.logstash:
  hosts: "${BEATS_HOST_PORT:log:5044}"
logging.level: warning

The drop fields was from Using AutoDiscover feature for Docker does not work when running in Swarm mode but that didn't work for me either.

My docker-compose file has the following service

  docker-beats:
    image: trajano.net/docker-beats
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    deploy:
      mode: global

Unfortunately I just get all the logs with no real attribution except for the file beats server