Docker.container.name is always empty

tonyskulk · March 6, 2019, 3:53am

Hi, I have a docker setup with filebeat and other containers producing logs that are fetched by the filebeat container. That works well so far, everything gets imported to elasticsearch indices and I can discover them in Kibana.

Now I want to filter all logs by docker.container.name but this field not set for any doc. I am using filebeat, kibana on version 6.4.3 and here is my filebeat.yml:

filebeat.inputs:
  - type: docker
    containers:
      ids: 
        - '*'
      path: "/usr/share/dockerlogs"

processors:
- decode_json_fields:
    fields: ["message"]
    target: ""
    overwrite_keys: true

- add_docker_metadata: ~

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

setup.kibana:
  host: "kibana:5601"

logging.to_files: true
logging.to_syslog: false

I thought that the processor - add_docker_metadata: ~ should do the trick, but unfortunately I am a bit lost now, cause it´s not working. Any help here?

Side note: All docker metadata fields (on https://www.elastic.co/guide/en/beats/filebeat/6.4/exported-fields-docker-processor.html) that should be passed are empty but docker.container.id

pierhugues · March 6, 2019, 3:16pm

By default it should connect to docker to retrieve that information, the processor logs a few information at debug level can you try to start filebeat with a debug log selector like this:

filebeat -v -e -d "add_docker_metadata"

tonyskulk · March 7, 2019, 5:20am

Thanks @pierhugues for your reply. I have enabled debug level for "add_docker_metadata" selector and got one one output line repeated >thousands of times like this:

{"log":"2019-03-06T15:33:41.845Z\u0009DEBUG\u0009[add_docker_metadata]\u0009add_docker_metadata/add_docker_metadata.go:186\u0009Container not found: cid=badb87a5b17a6493a0deacdd086aae713c904b3a2d511b9996e64f01b458b300-json.log\n","stream":"stderr","attrs":{"tag":"docker/elastic-filebeat"},"time":"2019-03-06T15:33:41.845724496Z"}

This line is repeated in the output maybe 100 times per second so I think there is something strange going on. I will try to get to debug log files again and will disable the log rotation to get the beginning logs.

So long I could only figure out the place where those log lines were produced. Seems like the last line of this:

container := d.watcher.Container(cid)
if container != nil {
	meta := common.MapStr{}
	metaIface, ok := event.Fields["docker"]
	if ok {
		meta = metaIface.(common.MapStr)
	}


	if len(container.Labels) > 0 {
		labels := common.MapStr{}
		for k, v := range container.Labels {
			safemapstr.Put(labels, k, v)
		}
		meta.Put("container.labels", labels)
	}


	meta.Put("container.id", container.ID)
	meta.Put("container.image", container.Image)
	meta.Put("container.name", container.Name)
	event.Fields["docker"] = meta.Clone()
} else {
	d.log.Debugf("Container not found: cid=%s", cid)
}

tonyskulk · March 7, 2019, 6:06am

Ok, now I have the full filebeat logs with debug outputs enabled for selector "add_docker_metadata":

March 7th 2019, 06:50:46.247	2019-03-07T05:50:46.245Z	INFO	instance/beat.go:565	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
March 7th 2019, 06:50:46.248	2019-03-07T05:50:46.248Z	INFO	instance/beat.go:572	Beat UUID: a54efd08-08ba-4845-839c-15e2f0793db1
March 7th 2019, 06:50:46.248	seccomp/seccomp.go:116	Syscall filter successfully installed
March 7th 2019, 06:50:46.248	instance/beat.go:789	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "a54efd08-08ba-4845-839c-15e2f0793db1"}}}
March 7th 2019, 06:50:46.248	instance/beat.go:798	Build info	{"system_info": {"build": {"commit": "a8afce82a3cbaffe30a9c23b5321a3183ae9dc1a", "libbeat": "6.4.3", "time": "2018-10-30T22:24:26.000Z", "version": "6.4.3"}}}
March 7th 2019, 06:50:46.248	instance/beat.go:801	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.10.3"}}}
March 7th 2019, 06:50:46.250	instance/beat.go:805	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-03-01T06:32:42Z","containerized":true,"hostname":"1e495f68e592","ips":["127.0.0.1/8","172.20.0.7/16"],"kernel_version":"4.4.0-1072-aws","mac_addresses":["02:42:ac:14:00:07"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":5,"patch":1804,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
March 7th 2019, 06:50:46.251	instance/beat.go:834	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2019-03-07T05:50:44.940Z"}}}
March 7th 2019, 06:50:46.251	2019-03-07T05:50:46.250Z	INFO	instance/beat.go:286	Setup Beat: filebeat; Version: 6.4.3
March 7th 2019, 06:50:46.258	2019-03-07T05:50:46.258Z	INFO	elasticsearch/client.go:163	Elasticsearch url: http://elasticsearch:9200
March 7th 2019, 06:50:46.259	2019-03-07T05:50:46.258Z	INFO	pipeline/module.go:98	Beat name: 1e495f68e592
March 7th 2019, 06:50:46.270	2019-03-07T05:50:46.270Z	INFO	instance/beat.go:383	filebeat start running.
March 7th 2019, 06:50:46.271	log/log.go:114	Starting metrics logging every 30s
March 7th 2019, 06:50:46.271	2019-03-07T05:50:46.271Z	INFO	registrar/registrar.go:134	Loading registrar data from /usr/share/filebeat/data/registry
March 7th 2019, 06:50:46.271	2019-03-07T05:50:46.271Z	INFO	registrar/registrar.go:141	States Loaded from registrar: 14
March 7th 2019, 06:50:46.271	2019-03-07T05:50:46.271Z	INFO	crawler/crawler.go:72	Loading Inputs: 1
March 7th 2019, 06:50:46.271	docker/input.go:46	EXPERIMENTAL: Docker input is enabled.
March 7th 2019, 06:50:46.310	2019-03-07T05:50:46.310Z	INFO	log/input.go:138	Configured paths: [/usr/share/dockerlogs/*/*.log]
March 7th 2019, 06:50:46.310	2019-03-07T05:50:46.310Z	INFO	input/input.go:114	Starting input of type: docker; ID: 1121490947606389615 
March 7th 2019, 06:50:46.310	2019-03-07T05:50:46.310Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
March 7th 2019, 06:50:46.314	2019-03-07T05:50:46.314Z	INFO	log/harvester.go:251	Harvester started for file: /usr/share/dockerlogs/1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09/1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.318	2019-03-07T05:50:46.317Z	INFO	log/harvester.go:251	Harvester started for file: /usr/share/dockerlogs/303f36227bbc62bd5e48b13b0009937078bff239e819595c31606c01fbf5ef24/303f36227bbc62bd5e48b13b0009937078bff239e819595c31606c01fbf5ef24-json.log
March 7th 2019, 06:50:46.323	add_docker_metadata/add_docker_metadata.go:186	Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.323	2019-03-07T05:50:46.323Z	INFO	log/harvester.go:251	Harvester started for file: /usr/share/dockerlogs/bc218a75eb08ac72eead153988b71dcd38d2d48ba5ac4f1796a1f7214691900e/bc218a75eb08ac72eead153988b71dcd38d2d48ba5ac4f1796a1f7214691900e-json.log
March 7th 2019, 06:50:46.323	add_docker_metadata/add_docker_metadata.go:186	Container not found: cid=303f36227bbc62bd5e48b13b0009937078bff239e819595c31606c01fbf5ef24-json.log
March 7th 2019, 06:50:46.324	add_docker_metadata/add_docker_metadata.go:186	Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.324	add_docker_metadata/add_docker_metadata.go:186	Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.324	add_docker_metadata/add_docker_metadata.go:186	Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log
March 7th 2019, 06:50:46.324	add_docker_metadata/add_docker_metadata.go:186	Container not found: cid=1e495f68e592be1131cce40f7d6cecd41eb9624d9e058b1f94ca8092827a5c09-json.log

The last lines continue for at least 10000+ lines.
I wonder why docker.container.id can be found while docker.container.name or other metadata fields can not. Feels like the container id is picked up another way than the other container metadata???
Along with this my docker-compose.yml could maybe be helpful:

version: "3"
services:
  filebeat:
    image: myimages/elastic-filebeat:latest
    container_name: elastic-filebeat
    volumes:
      - /var/lib/docker/containers:/usr/share/dockerlogs:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - filebeat_data:/usr/share/filebeat/data:rw
    user: root

volumes:
  filebeat_data:

networks:
  default:
    external:
      name: mynetwork

Maybe I have missed something, because it seems that this should be working for anyone else

tonyskulk · March 27, 2019, 3:09am

Still not able to get it running. Need help

system · April 24, 2019, 3:09am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.