Filebeat in docker doesn't send logs to logstash in another docker

bronweg · June 17, 2018, 8:33am

Hi all,

I'm newbie with elastic and also with docker.
I have created four containers with FileBeat, Logstash, Elasticsearch and Kibana.
I start FileBeat with -d "*" to see DEBUG logs and I see a lot of "inputs" but no "outputs" at all (no errors at all also).
If I enter to the container and run "ps aux | grep filebeat", I see process "filebeat -e" with PID 1.
If I'm running one more "filebeat -e" process, suddenly log records start to be sent to logstash and I can see them in Kibana.

Docker "run" command:

docker run --name "filebeat" --network "elk-network" \
-v "/root/ELK_on_DOCKER_with_ANSIBLE/roles/Filebeat/files/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro" \
-v "/var/lib/docker/containers:/hostfs/var/lib/docker/containers:ro" \
docker.elastic.co/beats/filebeat:6.3.0 -e -d "*"

filebeat.yml:

---
filebeat.inputs:
- type: log
  paths:
  - '/hostfs/var/lib/docker/containers/*/*.log'
output.logstash:
  hosts: ["logstash:5044"]

Any ideas?

adrisr · June 18, 2018, 7:43am

Can you paste the full debug log from filebeat?

bronweg · June 18, 2018, 8:03am

Debug log from Filebeat:

[root@localhost ELK_on_DOCKER_with_ANSIBLE]# docker run --name "filebeat" --network "elk-network" \
> -v "/root/ELK_on_DOCKER_with_ANSIBLE/roles/Filebeat/files/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro" \
> -v "/var/lib/docker/containers:/hostfs/var/lib/docker/containers:ro" \
> docker.elastic.co/beats/filebeat:6.3.0 -e -d "*"
2018-06-18T06:55:04.490Z	INFO	instance/beat.go:492	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2018-06-18T06:55:04.491Z	DEBUG	[beat]	instance/beat.go:519	Beat metadata path: /usr/share/filebeat/data/meta.json
2018-06-18T06:55:04.491Z	INFO	instance/beat.go:499	Beat UUID: d1571879-0289-48a8-9e6f-d3c8a4885556
2018-06-18T06:55:04.492Z	INFO	[beat]	instance/beat.go:716	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "d1571879-0289-48a8-9e6f-d3c8a4885556"}}}
2018-06-18T06:55:04.492Z	INFO	[beat]	instance/beat.go:725	Build info	{"system_info": {"build": {"commit": "a04cb664d5fbd4b1aab485d1766f3979c138fd38", "libbeat": "6.3.0", "time": "2018-06-11T22:34:44.000Z", "version": "6.3.0"}}}
2018-06-18T06:55:04.492Z	INFO	[beat]	instance/beat.go:728	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.9.4"}}}
2018-06-18T06:55:04.497Z	INFO	[beat]	instance/beat.go:732	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-06-17T06:10:53Z","containerized":true,"hostname":"59f89cee5118","ips":["127.0.0.1/8","172.18.0.5/16"],"kernel_version":"3.10.0-862.el7.x86_64","mac_addresses":["02:42:ac:12:00:05"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":5,"patch":1804,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2018-06-18T06:55:04.497Z	INFO	[beat]	instance/beat.go:761	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2018-06-18T06:55:03.070Z"}}}
2018-06-18T06:55:04.497Z	INFO	instance/beat.go:225	Setup Beat: filebeat; Version: 6.3.0
2018-06-18T06:55:04.498Z	DEBUG	[beat]	instance/beat.go:242	Initializing output plugins
2018-06-18T06:55:04.498Z	DEBUG	[processors]	processors/processor.go:49	Processors: 
2018-06-18T06:55:04.498Z	DEBUG	[publish]	pipeline/consumer.go:120	start pipeline event consumer
2018-06-18T06:55:04.498Z	INFO	pipeline/module.go:81	Beat name: 59f89cee5118
2018-06-18T06:55:04.499Z	INFO	instance/beat.go:315	filebeat start running.
2018-06-18T06:55:04.500Z	INFO	registrar/registrar.go:75	No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
2018-06-18T06:55:04.500Z	DEBUG	[registrar]	registrar/registrar.go:263	Write registry file: /usr/share/filebeat/data/registry
2018-06-18T06:55:04.502Z	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-06-18T06:55:04.506Z	DEBUG	[registrar]	registrar/registrar.go:290	Registry file updated. 0 states written.
2018-06-18T06:55:04.506Z	INFO	registrar/registrar.go:112	Loading registrar data from /usr/share/filebeat/data/registry
2018-06-18T06:55:04.506Z	INFO	registrar/registrar.go:123	States Loaded from registrar: 0
2018-06-18T06:55:04.506Z	WARN	beater/filebeat.go:354	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-06-18T06:55:04.506Z	INFO	crawler/crawler.go:48	Loading Inputs: 1
2018-06-18T06:55:04.507Z	DEBUG	[processors]	processors/processor.go:49	Processors: 
2018-06-18T06:55:04.507Z	DEBUG	[input]	log/config.go:178	recursive glob enabled
2018-06-18T06:55:04.507Z	DEBUG	[input]	log/input.go:120	exclude_files: []. Number of stats: 0
2018-06-18T06:55:04.507Z	DEBUG	[input]	log/input.go:141	input with previous states loaded: 0
2018-06-18T06:55:04.507Z	INFO	log/input.go:111	Configured paths: [/hostfs/var/lib/docker/containers/*/*.log]
2018-06-18T06:55:04.507Z	INFO	input/input.go:87	Starting input of type: log; ID: 17272755244748757091 
2018-06-18T06:55:04.507Z	INFO	crawler/crawler.go:82	Loading and starting Inputs completed. Enabled inputs: 1
2018-06-18T06:55:04.507Z	DEBUG	[registrar]	registrar/registrar.go:154	Starting Registrar
2018-06-18T06:55:04.507Z	DEBUG	[input]	log/input.go:147	Start next scan
2018-06-18T06:55:04.507Z	DEBUG	[input]	log/input.go:168	input states cleaned up. Before: 0, After: 0, Pending: 0
2018-06-18T06:55:14.508Z	DEBUG	[input]	input/input.go:124	Run input
2018-06-18T06:55:14.508Z	DEBUG	[input]	log/input.go:147	Start next scan
2018-06-18T06:55:14.508Z	DEBUG	[input]	log/input.go:168	input states cleaned up. Before: 0, After: 0, Pending: 0
2018-06-18T06:55:24.509Z	DEBUG	[input]	input/input.go:124	Run input
2018-06-18T06:55:24.509Z	DEBUG	[input]	log/input.go:147	Start next scan
2018-06-18T06:55:24.509Z	DEBUG	[input]	log/input.go:168	input states cleaned up. Before: 0, After: 0, Pending: 0
2018-06-18T06:55:34.511Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":61}},"total":{"ticks":110,"time":{"ms":112},"value":110},"user":{"ticks":50,"time":{"ms":51}}},"info":{"ephemeral_id":"dad3142a-9426-413f-99e3-5a6eb2bd6160","uptime":{"ms":30036}},"memstats":{"gc_next":4473924,"memory_alloc":2989464,"memory_total":2989464,"rss":12177408}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":1},"system":{"cpu":{"cores":1},"load":{"1":0.19,"15":0.59,"5":0.9,"norm":{"1":0.19,"15":0.59,"5":0.9}}}}}}
2018-06-18T06:55:34.514Z	DEBUG	[input]	input/input.go:124	Run input
2018-06-18T06:55:34.514Z	DEBUG	[input]	log/input.go:147	Start next scan
2018-06-18T06:55:34.514Z	DEBUG	[input]	log/input.go:168	input states cleaned up. Before: 0, After: 0, Pending: 0

adrisr · June 18, 2018, 9:25am

Hi,

Seems that filebeat is not finding any log files in /hostfs/var/lib/docker/containers. Can you check that your container has any .log files in there? If there is a problem with the bind mount the directory will be empty.

bronweg · June 18, 2018, 4:10pm

No, there are no issues with mount. More than that, if I run "filebeat -e" second time in the container (manually) I can see the logs in Kibana.

Bellow is output from the container, so we can see that logs are there:

[root@filebeat containers]# pwd
/hostfs/var/lib/docker/containers
[root@filebeat containers]# ls -l */*.log
-rw-r-----. 1 root root 11134 Jun 18 16:06 2fc19d25fd3472b59eeb65929222a8601dbd972146419f5c47c192c6968cbc37/2fc19d25fd3472b59eeb65929222a8601dbd972146419f5c47c192c6968cbc37-json.log
-rw-r-----. 1 root root     0 Jun 18 16:05 4f20b54080dea30e393aa464602aba4e992f8a3c386a1aaa9a36d36a76fe6b39/4f20b54080dea30e393aa464602aba4e992f8a3c386a1aaa9a36d36a76fe6b39-json.log
-rw-r-----. 1 root root   270 Jun 18 16:05 c7715098ca842cbe7eadd19d32e9b4248d2e11ecf55030d9b7c801247b5e51e8/c7715098ca842cbe7eadd19d32e9b4248d2e11ecf55030d9b7c801247b5e51e8-json.log
-rw-r-----. 1 root root  7337 Jun 18 16:06 d3b838e69edcc74a560373a9c875996bc5609d2312ab625bea37d37d4620940a/d3b838e69edcc74a560373a9c875996bc5609d2312ab625bea37d37d4620940a-json.log
[root@filebeat containers]#

bronweg · June 21, 2018, 4:09am

Any ideas?
I am still on this issue.

system · July 19, 2018, 4:09am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.