Missing docker metadata in filebeat

Elastic Stack Beats
,
1

Hi all,

Somebody experiencing the same problem that filebeat doesn't send any docker metadata for the docker container logging?

Symptom:
Since docker version 19.x you need to configure "type=container" as filebeat.input.

 filebeat.prospectors:  
    - type: container  
	  enabled: true  
      paths:  
        - /var/lib/docker/containers/*/*.log  
    processors:  
      - add_docker_metadata: ~

Filebeat will sends this output from a RedHat system that runs docker version 19.03.1:

    "container": {
      "id": "containers"
    },

so the container metadata doesn't have the appropriate docker id, name, image name and labels information's as shown in the example below from system that runs docker version 18.06.3 with the appropriate filebeat input config:

 filebeat.prospectors:  
    - type: docker  
        enabled: true  
	    containers.ids:	'*'  
        container.paths:  
          - /var/lib/docker/containers/*/*.log  
    processors:  
      - add_docker_metadata: ~

Output looks like:

    "container": {  
      "name": "[docker_name]",  
      "image": {  
        "name": "[docker_image]"  
      },  
      "id": "[docker_id]",  
      "labels": {  
        "com_docker_swarm_task_name": "[docker_swarm_id]",  
        "com_docker_swarm_task": "",  
        "MAINTAINER": "[Maintainer]",  
        "com_docker_swarm_service_name": "[docker_swarm_service_name]",  
        "com_docker_swarm_task_id": "[swarm_id]",  
        "com_docker_swarm_service_id": "[swarm_sid]",  
        "git-commit": "55d67d5",  
        "logger": "[logger_id]",  
        "com_docker_swarm_node_id": "[swarm_node_id]",  
        "com_docker_stack_namespace": "[swarm_namepsace]"  
      }  
    },

Any help will be appreciated.
Thanks in advance.

Regards,

Eihkoh

2

Did you try to switch to the DEBUG log level?

3

Hi mtojek,

I use logging.level=error but don't see any error related to this.
However when I enable the DEBUG level I see the following messages:

May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.910+0200        DEBUG        [docker]        docker/client.go:48        Docker client will negotiate the API version on the first request.
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.942+0200        DEBUG        [add_docker_metadata]        add_docker_metadata/add_docker_metadata.go:91        add_docker_metadata: docker environment detected
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.942+0200        DEBUG        [docker]        docker/watcher.go:198        Start docker containers scanner
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.942+0200        DEBUG        [docker]        docker/watcher.go:333        List containers
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200        DEBUG        [bus]        bus/bus.go:83        docker: map[container:0xc000554000 start:true]
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200        DEBUG        [docker]        docker/watcher.go:246        Fetching events since 1589812692
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200        DEBUG        [bus]        bus/bus.go:83        docker: map[container:0xc000554070 start:true]
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200        DEBUG        [processors]        processors/processor.go:101        Generated new processors: add_host_metadata=[netinfo.enabled=[false], cache.ttl=[5m0s]], add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.ppid]]
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200        DEBUG        [bus]        bus/bus.go:83        docker: map[container:0xc0005540e0 start:true]
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200        DEBUG        [bus]        bus/bus.go:83        docker: map[container:0xc000554150 start:true]
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200        DEBUG        [bus]        bus/bus.go:83        docker: map[container:0xc0005541c0 start:true]
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200        DEBUG        [bus]        bus/bus.go:83        docker: map[container:0xc000554230 start:true]

So filebeat detects the log files of running containers. However when the file start to harvest you get that the cid is not found during the add_docker_metadata processing.. the log looks as follow:

May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.953+0200        DEBUG        [input]        log/input.go:421        Check file for harvesting: /data/var/lib/docker/containers/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a-json.log
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.953+0200        DEBUG        [input]        log/input.go:494        Start harvester for new file: /data/var/lib/docker/containers/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a-json.log
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.953+0200        DEBUG        [add_docker_metadata]        add_docker_metadata/add_docker_metadata.go:207        Container not found: cid=containers
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200        DEBUG        [processors]        processing/processors.go:186        Publish event: {
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200        DEBUG        [harvester]        log/harvester.go:501        Setting offset for file based on seek: /data/var/lib/docker/containers/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a-json.log
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200        DEBUG        [add_docker_metadata]        add_docker_metadata/add_docker_metadata.go:207        Container not found: cid=containers
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200        DEBUG        [harvester]        log/harvester.go:487        Setting offset for file: /data/var/lib/docker/containers/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a-json.log. Offset: 0
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200        DEBUG        [harvester]        log/harvester.go:182        Harvester setup successful. Line terminator: 1
May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200        DEBUG        [processors]        processing/processors.go:186        Publish event: {
... etc

Looks like that cid filed is not fill with the proper docker container id?

Regards,

eihkoh

4

You have an additional folder in the path (/data). Usually it's installed in the /var/lib/docker/.... Navigate to the documentation and check the match_source_index option: https://www.elastic.co/guide/en/beats/filebeat/master/add-docker-metadata.html

5

Hi mjotek,

Indeed.. I saw this morning the same url ! shame me..
I will proceed to test with using match_source_index.. Will update it.

Regards,

eihkoh

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.