Hi,
I have several docker containers (using docker-compose) and I'm trying to use Filebeat (from docker) to push my containers logs to elasticsearch with no success.
This is my filebeat.yml:
filebeat.inputs:
- type: docker
containers.ids: '*'
output.elasticsearch:
hosts: ["outflink-elastic:9200"]
protocol: "http"
This is its config block in docker-compose.yml:
filebeat:
image: akapit/outflink_platform:outflink-filebeat
container_name: outflink-filebeat
user: root
volumes:
- $HOME/filebeat_data:/usr/share/filebeat/data
- /var/lib/docker/containers:/usr/share/filebeat/dockerlogs:ro
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- elasticsearch
This is my Filebeat Dockerfile:
FROM docker.elastic.co/beats/filebeat:6.5.4
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
RUN mkdir /usr/share/filebeat/dockerlogs
USER filebeat
This is the filebeat console output:
2019-01-24T10:09:42.348Z INFO instance/beat.go:592 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-01-24T10:09:42.354Z INFO instance/beat.go:599 Beat UUID: 7e44573c-940f-4ea6-94ab-2673f9962e6c
2019-01-24T10:09:42.354Z INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
2019-01-24T10:09:42.354Z INFO [beat] instance/beat.go:825 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "7e44573c-940f-4ea6-94ab-2673f9962e6c"}}}
2019-01-24T10:09:42.356Z INFO [beat] instance/beat.go:834 Build info {"system_info": {"build": {"commit": "bd8922f1c7e93d12b07e0b3f7d349e17107f7826", "libbeat": "6.5.4", "time": "2018-12-17T20:22:29.000Z", "version": "6.5.4"}}}
2019-01-24T10:09:42.356Z INFO [beat] instance/beat.go:837 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.6"}}}
2019-01-24T10:09:42.363Z INFO [beat] instance/beat.go:841 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-01-24T04:34:57Z","containerized":true,"name":"92a6c0af3b08","ip":["127.0.0.1/8","192.168.96.7/20"],"kernel_version":"4.9.125-linuxkit","mac":["02:42:c0:a8:60:07"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-01-24T10:09:42.364Z INFO [beat] instance/beat.go:870 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2019-01-24T10:09:39.920Z"}}}
2019-01-24T10:09:42.365Z INFO instance/beat.go:278 Setup Beat: filebeat; Version: 6.5.4
2019-01-24T10:09:42.366Z INFO elasticsearch/client.go:163 Elasticsearch url: http://outflink-elastic:9200
2019-01-24T10:09:42.367Z INFO [publisher] pipeline/module.go:110 Beat name: 92a6c0af3b08
2019-01-24T10:09:42.442Z INFO instance/beat.go:400 filebeat start running.
2019-01-24T10:09:42.443Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2019-01-24T10:09:42.448Z INFO registrar/registrar.go:134 Loading registrar data from /usr/share/filebeat/data/registry
2019-01-24T10:09:42.451Z INFO registrar/registrar.go:141 States Loaded from registrar: 36
2019-01-24T10:09:42.452Z INFO crawler/crawler.go:72 Loading Inputs: 1
2019-01-24T10:09:42.630Z INFO log/input.go:138 Configured paths: [/var/lib/docker/containers/*/*.log]
2019-01-24T10:09:42.630Z INFO input/input.go:114 Starting input of type: docker; ID: 8783912846562670036
2019-01-24T10:09:42.630Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-01-24T10:10:12.486Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":260,"time":{"ms":260}},"total":{"ticks":330,"time":{"ms":330},"value":330},"user":{"ticks":70,"time":{"ms":70}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"cab0928f-44f4-4838-ab5c-d0ab24c8818d","uptime":{"ms":30310}},"memstats":{"gc_next":4194304,"memory_alloc":1821080,"memory_total":4872672,"rss":9252864}},"filebeat":{"events":{"added":56,"done":56},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0,"filtered":56,"total":56}}},"registrar":{"states":{"cleanup":28,"current":8,"update":56},"writes":{"success":56,"total":56}},"system":{"cpu":{"cores":4},"load":{"1":3.25,"15":1.42,"5":1.42,"norm":{"1":0.8125,"15":0.355,"5":0.355}}}}}}
As you can see, it doesn't look that there is an error, however nothing gets written in elastic.
I'd appreciate any help,
Thanks!