Filebeat 6.3.2 doesn´t detect every started container

I´m using filebeat 6.3.2 to ship container logs.
This is my current filebeat configuration (see below).

---
filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.name: dwp-
          config:
            - type: log
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              json.keys_under_root: true
              fields:
                level: info
                labels: ${data.docker.container.labels.label1}
name: filebeat
logging.level: info
fields_under_root: true
processors:
- decode_json_fields:
    fields: ["log"]
    target: "mslog2"
- rename:
    fields:
      - from: "log"
        to: "mslog"
path.data: /filebeat/data
filebeat.registry_file: ${path.data}/myregistry
output.console:
  enabled: true
  pretty: true

When I start containers, then some containers are detected and some aren´t. In the example below, the first container 593c3fe4666ddf0c1c02fb6b9fac290da49ebdc5a137b35aed09f6fb23830b5c is detected and logfiles are shipped, the second container with id 3f98fe64b36e014545bf038fc1e077454bb67787e9935b1e5114483fdd371d52 is not detected.

Start of Containers. First Container is detected, second container isn´t.

docker run  --name dwp-mno  --log-driver json-file   --log-opt max-size=2m --log-opt max-file=2   -d   centos  sh -c 'for ((j=1; j<=3000; j++)) do for ((i=1; i<=10; i++)) ; do echo " {w: $j }"; sleep 0.01;   done;  done'
593c3fe4666ddf0c1c02fb6b9fac290da49ebdc5a137b35aed09f6fb23830b5c

docker run  --name dwp-pqr  --log-driver json-file   --log-opt max-size=2m --log-opt max-file=2   -d   centos  sh -c 'for ((j=1; j<=3000; j++)) do for ((i=1; i<=10; i++)) ; do echo " {w: $j }"; sleep 0.01;   done;  done'
3f98fe64b36e014545bf038fc1e077454bb67787e9935b1e5114483fdd371d52

filebeat output

Aug 30 09:22:28 xxx docker[52630]: 2018-08-30T09:22:28.250+0200        INFO        log/input.go:118        Configured paths: [/var/lib/docker/containers/**593c3fe4666ddf0c1c02fb6b9fac290da49ebdc5a137b35aed09f6fb23830b5c**/*.log]
Aug 30 09:22:28 xxx docker[52630]: 2018-08-30T09:22:28.250+0200        INFO        autodiscover/autodiscover.go:144        Autodiscover starting runner: input [type=log, ID=5438576869600517932]
Aug 30 09:22:28 xxx docker[52630]: 2018-08-30T09:22:28.250+0200        INFO        input/input.go:88        Starting input of type: log; ID: 5438576869600517932
Aug 30 09:22:28 xxx docker[52630]: 2018-08-30T09:22:28.251+0200        INFO        log/harvester.go:228        Harvester started for file: /var/lib/docker/containers/**593c3fe4666ddf0c1c02fb6b9fac290da49ebdc5a137b35aed09f6fb23830b5c**/593c3fe4666ddf0c1c02fb6b9fac290da49ebdc5a137b35aed09f6fb23830b5c-json.log
Aug 30 09:22:55 xxx docker[52630]: 2018-08-30T09:22:55.009+0200        INFO        [monitoring]        log/log.go:124        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":137540,"time":{"ms":559}},"total":{"ticks":248990,"time":{"ms":1028},"value":248990},"user":{"ticks":111450,"time":{"ms":469}}},"info":{"ephemeral_id":"2d8784f0-bf6b-4658-8699-a0499df8838d","uptime":{"ms":7350035}},"memstats":{"gc_next":57799152,"memory_alloc":29039104,"memory_total":17526160320}},"filebeat":{"events":{"active":-52,"added":3241,"done":3293},"harvester":{"open_files":7,"running":7,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":3292,"batches":30,"total":3292},"read":{"bytes":1047},"write":{"bytes":2788639}},"pipeline":{"clients":107,"events":{"active":34,"filtered":1,"published":3263,"total":3263},"queue":{"acked":3292}}},"registrar":{"states":{"current":71,"update":3293},"writes":{"success":31,"total":31}},"system":{"load":{"1":5.32,"15":1.68,"5":2.61,"norm":{"1":0.665,"15":0.21,"5":0.3263}}}}}}

Is this a know issue/is something wrong with my configuration

Try this configuration:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: dwp-
          config:
            - type: docker
              containers:
                path: '/var/lib/docker/containers/'
                stream: 'all'
                ids:
                  - '${data.docker.container.id}'

output.console:
  pretty: true
1 Like

I tried it like that, but now filebeat doesn´t find any docker containers (logs attached)
I added the docker run filebeat command to show that filebeat has access to /var/lib/docker/containers/ and /var/run/docker.sock

Logs without any trace of detected container logs.

-- Logs begin at Thu 2018-08-30 11:13:50 CEST. --
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.526+0200        INFO        instance/beat.go:315        filebeat start running.
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.526+0200        INFO        registrar/registrar.go:80        No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.528+0200        INFO        [monitoring]        log/log.go:97        Starting metrics logging every 30s
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.529+0200        INFO        registrar/registrar.go:117        Loading registrar data from /usr/share/filebeat/data/registry
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.529+0200        INFO        registrar/registrar.go:124        States Loaded from registrar: 0
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.529+0200        WARN        beater/filebeat.go:354        Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.529+0200        INFO        crawler/crawler.go:48        Loading Inputs: 0
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.529+0200        INFO        crawler/crawler.go:82        Loading and starting Inputs completed. Enabled inputs: 0
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.529+0200        WARN        [cfgwarn]        docker/docker.go:34        BETA: The docker autodiscover is beta
Aug 30 12:17:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:21.556+0200        INFO        autodiscover/autodiscover.go:76        Starting autodiscover manager
Aug 30 12:17:51 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:17:51.532+0200        INFO        [monitoring]        log/log.go:124        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":142}},"total":{"ticks":160,"time":{"ms":167},"value":160},"user":{"ticks":20,"time":{"ms":25}}},"info":{"ephemeral_id":"be2b3db3-17b6-441e-ad38-2f2702dc875c","uptime":{"ms":30015}},"memstats":{"gc_next":4194304,"memory_alloc":3021376,"memory_total":4797272,"rss":15699968}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"console"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":8},"load":{"1":1.02,"15":1.2,"5":1.06,"norm":{"1":0.1275,"15":0.15,"5":0.1325}}}}}}
Aug 30 12:18:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:18:21.530+0200        INFO        [monitoring]        log/log.go:124        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":2}},"total":{"ticks":160,"time":{"ms":3},"value":160},"user":{"ticks":20,"time":{"ms":1}}},"info":{"ephemeral_id":"be2b3db3-17b6-441e-ad38-2f2702dc875c","uptime":{"ms":60015}},"memstats":{"gc_next":4194304,"memory_alloc":3222784,"memory_total":4998680}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.68,"15":1.17,"5":0.98,"norm":{"1":0.085,"15":0.1463,"5":0.1225}}}}}}
Aug 30 12:18:51 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:18:51.530+0200        INFO        [monitoring]        log/log.go:124        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":3}},"total":{"ticks":160,"time":{"ms":3},"value":160},"user":{"ticks":20}},"info":{"ephemeral_id":"be2b3db3-17b6-441e-ad38-2f2702dc875c","uptime":{"ms":90015}},"memstats":{"gc_next":4194304,"memory_alloc":3397408,"memory_total":5173304}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.56,"15":1.14,"5":0.92,"norm":{"1":0.07,"15":0.1425,"5":0.115}}}}}}
Aug 30 12:19:21 dwpmicrappe04.admnet.fits docker[4183]: 2018-08-30T12:19:21.532+0200        INFO        [monitoring]        log/log.go:124        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":8}},"total":{"ticks":170,"time":{"ms":9},"value":170},"user":{"ticks":20,"time":{"ms":1}}},"info":{"ephemeral_id":"be2b3db3-17b6-441e-ad38-2f2702dc875c","uptime":{"ms":120015}},"memstats":{"gc_next":4194304,"memory_alloc":1721248,"memory_total":5385016,"rss":233472}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.5,"15":1.11,"5":0.86,"norm":{"1":0.0625,"15":0.1388,"5":0.1075}}}}}}

docker run for the container producing the output:

docker run  --name dwp-vwx  --log-driver json-file   --log-opt max-size=2m --log-opt max-file=2   -d   centos  sh -c 'for ((j=1; j<=30; j++)) do for ((i=1; i<=10; i++)) ; do echo " {w: $j }"; sleep 0.1;   done;  done'

docker run of filebeat

/usr/bin/docker run --name=dwp-filebeat \
    --privileged=true \
    --log-driver json-file \
    --log-opt max-size=200m \
    --log-opt max-file=2 \
    --user 0 \
    --label co.elastic.logs/disable=true \
    --add-host "endpoint1:2.104.19.25" \
    --add-host "endpoint2:2.104.19.26" \
    --add-host "endpoint3:2.104.19.27" \
    --add-host "endpoint4:2.104.19.28" \
    --volume /appdata/container/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml:z \
    --volume /appdata/container/filebeat/data:/filebeat/data:z \
    --volume /var/lib/docker:/var/lib/docker \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    --env "TZ=Europe/Berlin" \
    docker.elastic.co/beats/filebeat:6.3.2

There was one typo in the config that I overlooked.

I changed it as described below and I´m testing again.

From:

contains:
              docker.container.image: dwp-

To,

contains:
              docker.container.name: dwp-

is couldn´t find a complete spec for filebeat.aotdiscover for docker. (filebeat 6.3)

Are there any recommendations besides https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html

Thank you very much, I cannot reproduce the error with your config.

Because you're running filebeat in a docker container and containers logs are stored in this path /var/lib/docker/containers/ on the host but this path is not visible inside the filebeat container. So you need to mount the host directory when running filbeat:

-v /var/lib/docker/containers:/usr/share/filebeat/containers

Now, the above path is visible as defined in the provided config:

              ...
              containers:
                path: '/var/lib/docker/containers/'
                stream: 'all'
              ...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.