Ok, now I have the full filebeat logs with debug outputs enabled for selector "add_docker_metadata":
March 7th 2019, 06:50:46.247 2019-03-07T05:50:46.245Z INFO instance/beat.go:565 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
March 7th 2019, 06:50:46.248 2019-03-07T05:50:46.248Z INFO instance/beat.go:572 Beat UUID: a54efd08-08ba-4845-839c-15e2f0793db1
March 7th 2019, 06:50:46.248 seccomp/seccomp.go:116 Syscall filter successfully installed
March 7th 2019, 06:50:46.248 instance/beat.go:789 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "a54efd08-08ba-4845-839c-15e2f0793db1"}}}
March 7th 2019, 06:50:46.248 instance/beat.go:798 Build info {"system_info": {"build": {"commit": "a8afce82a3cbaffe30a9c23b5321a3183ae9dc1a", "libbeat": "6.4.3", "time": "2018-10-30T22:24:26.000Z", "version": "6.4.3"}}}
March 7th 2019, 06:50:46.248 instance/beat.go:801 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.10.3"}}}
March 7th 2019, 06:50:46.250 instance/beat.go:805 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-03-01T06:32:42Z","containerized":true,"hostname":"1e495f68e592","ips":["127.0.0.1/8","172.20.0.7/16"],"kernel_version":"4.4.0-1072-aws","mac_addresses":["02:42:ac:14:00:07"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":5,"patch":1804,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
March 7th 2019, 06:50:46.251 instance/beat.go:834 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2019-03-07T05:50:44.940Z"}}}
March 7th 2019, 06:50:46.251 2019-03-07T05:50:46.250Z INFO instance/beat.go:286 Setup Beat: filebeat; Version: 6.4.3
March 7th 2019, 06:50:46.258 2019-03-07T05:50:46.258Z INFO elasticsearch/client.go:163 Elasticsearch url: http://elasticsearch:9200
March 7th 2019, 06:50:46.259 2019-03-07T05:50:46.258Z INFO pipeline/module.go:98 Beat name: 1e495f68e592
March 7th 2019, 06:50:46.270 2019-03-07T05:50:46.270Z INFO instance/beat.go:383 filebeat start running.
March 7th 2019, 06:50:46.271 log/log.go:114 Starting metrics logging every 30s
March 7th 2019, 06:50:46.271 2019-03-07T05:50:46.271Z INFO registrar/registrar.go:134 Loading registrar data from /usr/share/filebeat/data/registry
March 7th 2019, 06:50:46.271 2019-03-07T05:50:46.271Z INFO registrar/registrar.go:141 States Loaded from registrar: 14
March 7th 2019, 06:50:46.271 2019-03-07T05:50:46.271Z INFO crawler/crawler.go:72 Loading Inputs: 1
March 7th 2019, 06:50:46.271 docker/input.go:46 EXPERIMENTAL: Docker input is enabled.
March 7th 2019, 06:50:46.310 2019-03-07T05:50:46.310Z INFO log/input.go:138 Configured paths: [/usr/share/dockerlogs/*/*.log]
March 7th 2019, 06:50:46.310 2019-03-07T05:50:46.310Z INFO input/input.go:114 Starting input of type: docker; ID: 1121490947606389615
March 7th 2019, 06:50:46.310 2019-03-07T05:50:46.310Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
March 7th 2019, 06:50:46.314 2019-03-07T05:50:46.314Z INFO log/harvester.go:251 Harvester started for file: /usr/share/dockerlogs/1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09/1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.318 2019-03-07T05:50:46.317Z INFO log/harvester.go:251 Harvester started for file: /usr/share/dockerlogs/303f36227bbc62bd5e48b13b0009937078bff239e819595c31606c01fbf5ef24/303f36227bbc62bd5e48b13b0009937078bff239e819595c31606c01fbf5ef24-json.log
March 7th 2019, 06:50:46.323 add_docker_metadata/add_docker_metadata.go:186 Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.323 2019-03-07T05:50:46.323Z INFO log/harvester.go:251 Harvester started for file: /usr/share/dockerlogs/bc218a75eb08ac72eead153988b71dcd38d2d48ba5ac4f1796a1f7214691900e/bc218a75eb08ac72eead153988b71dcd38d2d48ba5ac4f1796a1f7214691900e-json.log
March 7th 2019, 06:50:46.323 add_docker_metadata/add_docker_metadata.go:186 Container not found: cid=303f36227bbc62bd5e48b13b0009937078bff239e819595c31606c01fbf5ef24-json.log
March 7th 2019, 06:50:46.324 add_docker_metadata/add_docker_metadata.go:186 Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.324 add_docker_metadata/add_docker_metadata.go:186 Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.324 add_docker_metadata/add_docker_metadata.go:186 Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.324 add_docker_metadata/add_docker_metadata.go:186 Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
The last lines continue for at least 10000+ lines.
I wonder why docker.container.id can be found while docker.container.name or other metadata fields can not. Feels like the container id is picked up another way than the other container metadata???
Along with this my docker-compose.yml could maybe be helpful:
version: "3"
services:
filebeat:
image: myimages/elastic-filebeat:latest
container_name: elastic-filebeat
volumes:
- /var/lib/docker/containers:/usr/share/dockerlogs:ro
- /var/run/docker.sock:/var/run/docker.sock
- filebeat_data:/usr/share/filebeat/data:rw
user: root
volumes:
filebeat_data:
networks:
default:
external:
name: mynetwork
Maybe I have missed something, because it seems that this should be working for anyone else