# Missing docker metadata in filebeat **URL:** \ **Category:** Beats\ **Tags:** docker, filebeat\ **Created:** [May 18, 2020, 11:29am UTC](https://discuss.elastic.co/t/missing-docker-metadata-in-filebeat/233095 "2020-05-18T11:29:56Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![eihkoh](https://avatars.discourse-cdn.com/v4/letter/e/2acd7d/32.png) [@eihkoh](https://discuss.elastic.co/u/eihkoh)\ **Post date:** [May 18, 2020, 11:29am UTC](https://discuss.elastic.co/t/missing-docker-metadata-in-filebeat/233095/1 "2020-05-18T11:29:56Z")
Hi all, Somebody experiencing the same problem that filebeat doesn't send any docker metadata for the docker container logging? **Symptom:** Since docker version _19.x_ you need to configure _"type=container"_ as filebeat.input. ```auto filebeat.prospectors: - type: container enabled: true paths: - /var/lib/docker/containers/*/*.log processors: - add_docker_metadata: ~ ``` Filebeat will sends this output from a RedHat system that runs docker version _19.03.1_: ```auto "container": { "id": "containers" }, ``` so the container metadata doesn't have the appropriate docker id, name, image name and labels information's as shown in the example below from system that runs docker version _18.06.3_ with the appropriate filebeat input config: ```auto filebeat.prospectors: - type: docker enabled: true containers.ids: '*' container.paths: - /var/lib/docker/containers/*/*.log processors: - add_docker_metadata: ~ ``` Output looks like: ```auto "container": { "name": "[docker_name]", "image": { "name": "[docker_image]" }, "id": "[docker_id]", "labels": { "com_docker_swarm_task_name": "[docker_swarm_id]", "com_docker_swarm_task": "", "MAINTAINER": "[Maintainer]", "com_docker_swarm_service_name": "[docker_swarm_service_name]", "com_docker_swarm_task_id": "[swarm_id]", "com_docker_swarm_service_id": "[swarm_sid]", "git-commit": "55d67d5", "logger": "[logger_id]", "com_docker_swarm_node_id": "[swarm_node_id]", "com_docker_stack_namespace": "[swarm_namepsace]" } }, ``` Any help will be appreciated. Thanks in advance. Regards, Eihkoh ---
**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\ **Post date:** [May 18, 2020, 1:50pm UTC](https://discuss.elastic.co/t/missing-docker-metadata-in-filebeat/233095/2 "2020-05-18T13:50:20Z")
Did you try to switch to the DEBUG log level? ---
**Author:** ![eihkoh](https://avatars.discourse-cdn.com/v4/letter/e/2acd7d/32.png) [@eihkoh](https://discuss.elastic.co/u/eihkoh)\ **Post date:** [May 19, 2020, 7:27am UTC](https://discuss.elastic.co/t/missing-docker-metadata-in-filebeat/233095/3 "2020-05-19T07:27:59Z")
Hi mtojek, I use logging.level=error but don't see any error related to this. However when I enable the DEBUG level I see the following messages: ```auto May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.910+0200 DEBUG [docker] docker/client.go:48 Docker client will negotiate the API version on the first request. May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.942+0200 DEBUG [add_docker_metadata] add_docker_metadata/add_docker_metadata.go:91 add_docker_metadata: docker environment detected May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.942+0200 DEBUG [docker] docker/watcher.go:198 Start docker containers scanner May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.942+0200 DEBUG [docker] docker/watcher.go:333 List containers May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200 DEBUG [bus] bus/bus.go:83 docker: map[container:0xc000554000 start:true] May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200 DEBUG [docker] docker/watcher.go:246 Fetching events since 1589812692 May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200 DEBUG [bus] bus/bus.go:83 docker: map[container:0xc000554070 start:true] May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200 DEBUG [processors] processors/processor.go:101 Generated new processors: add_host_metadata=[netinfo.enabled=[false], cache.ttl=[5m0s]], add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.ppid]] May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200 DEBUG [bus] bus/bus.go:83 docker: map[container:0xc0005540e0 start:true] May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200 DEBUG [bus] bus/bus.go:83 docker: map[container:0xc000554150 start:true] May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200 DEBUG [bus] bus/bus.go:83 docker: map[container:0xc0005541c0 start:true] May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.946+0200 DEBUG [bus] bus/bus.go:83 docker: map[container:0xc000554230 start:true] ``` So filebeat detects the log files of running containers. However when the file start to harvest you get that the _cid_ is not found during the _add\_docker\_metadata_ processing.. the log looks as follow: ```auto May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.953+0200 DEBUG [input] log/input.go:421 Check file for harvesting: /data/var/lib/docker/containers/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a-json.log May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.953+0200 DEBUG [input] log/input.go:494 Start harvester for new file: /data/var/lib/docker/containers/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a-json.log May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.953+0200 DEBUG [add_docker_metadata] add_docker_metadata/add_docker_metadata.go:207 Container not found: cid=containers May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200 DEBUG [processors] processing/processors.go:186 Publish event: { May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200 DEBUG [harvester] log/harvester.go:501 Setting offset for file based on seek: /data/var/lib/docker/containers/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a-json.log May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200 DEBUG [add_docker_metadata] add_docker_metadata/add_docker_metadata.go:207 Container not found: cid=containers May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200 DEBUG [harvester] log/harvester.go:487 Setting offset for file: /data/var/lib/docker/containers/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a/8ecd6d7e294e5a5d868c176a6ca1f189c0c35450330cbfbfbab8171200c0554a-json.log. Offset: 0 May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200 DEBUG [harvester] log/harvester.go:182 Harvester setup successful. Line terminator: 1 May 18 16:38:12 [hostname] filebeat[45566]: 2020-05-18T16:38:12.954+0200 DEBUG [processors] processing/processors.go:186 Publish event: { ... etc ``` Looks like that _cid_ filed is not fill with the proper docker container id? Regards, eihkoh ---
**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\ **Post date:** [May 19, 2020, 7:53am UTC](https://discuss.elastic.co/t/missing-docker-metadata-in-filebeat/233095/4 "2020-05-19T07:53:27Z")
You have an additional folder in the path (`/data`). Usually it's installed in the `/var/lib/docker/...`. Navigate to the documentation and check the `match_source_index` option: [https://www.elastic.co/guide/en/beats/filebeat/master/add-docker-metadata.html](https://www.elastic.co/guide/en/beats/filebeat/master/add-docker-metadata.html) ---
**Author:** ![eihkoh](https://avatars.discourse-cdn.com/v4/letter/e/2acd7d/32.png) [@eihkoh](https://discuss.elastic.co/u/eihkoh)\ **Post date:** [May 19, 2020, 8:11am UTC](https://discuss.elastic.co/t/missing-docker-metadata-in-filebeat/233095/5 "2020-05-19T08:11:28Z")
Hi mjotek, Indeed.. I saw this morning the same url ! shame me.. I will proceed to test with using _match\_source\_index_.. Will update it. Regards, eihkoh ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 16, 2020, 8:11am UTC](https://discuss.elastic.co/t/missing-docker-metadata-in-filebeat/233095/6 "2020-06-16T08:11:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.