I'm trying to use filebeat docker but it's not harvesting files.
Autodiscover seems to work because found the logs I want but it's not harvesting them.
This is my filebeat.yml:
filebeat.autodiscover:
providers:
- type: docker templates: - condition: contains: docker.container.image: nginx config: - type: container paths: - /var/lib/docker/containers/${data.docker.container.id}/*.log exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart linesoutput.elasticsearch:
hosts: '192.168.141.109:30920'
and my output log:
2021-02-05T13:48:23.936Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2021-02-05T13:48:23.939Z INFO instance/beat.go:653 Beat ID: 8531f8b3-2e5e-4b76-b228-ac638bc39a69
2021-02-05T13:48:23.940Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-02-05T13:48:23.940Z INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "8531f8b3-2e5e-4b76-b228-ac638bc39a69"}}}
2021-02-05T13:48:23.940Z INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T23:11:24.000Z", "version": "7.10.2"}}}
2021-02-05T13:48:23.940Z INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.12"}}}
2021-02-05T13:48:23.943Z INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-05T10:36:38Z","containerized":true,"name":"247dcdf397d7","ip":["127.0.0.1/8","172.17.0.3/16"],"kernel_version":"5.4.0-53-generic","mac":["02:42:ac:11:00:03"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"3d458769eadb4ca1ac689c5da40af2bf"}}}
2021-02-05T13:48:23.944Z INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 20, "ppid": 19, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-02-05T13:48:23.400Z"}}}
2021-02-05T13:48:23.944Z INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.10.2
2021-02-05T13:48:23.944Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.10.2' as ILM is enabled.
2021-02-05T13:48:23.944Z INFO eslegclient/connection.go:99 elasticsearch url: http://192.168.141.109:30920
2021-02-05T13:48:23.945Z INFO [publisher] pipeline/module.go:113 Beat name: 247dcdf397d7
2021-02-05T13:48:23.947Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-02-05T13:48:23.947Z INFO instance/beat.go:455 filebeat start running.
2021-02-05T13:48:23.958Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-02-05T13:48:23.959Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2021-02-05T13:48:23.960Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2021-02-05T13:48:23.961Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 0
2021-02-05T13:48:23.961Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0
2021-02-05T13:48:24.002Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager
2021-02-05T13:48:24.004Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/67e21b763a8c830c504933afb302c2c1237997b5883f2cfd69d75cf37ff254c7/.log]
2021-02-05T13:48:24.005Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/67e21b763a8c830c504933afb302c2c1237997b5883f2cfd69d75cf37ff254c7/.log]
2021-02-05T13:48:53.965Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"id":"247dcdf397d729708a6c44e2113988babd67fdb774dc6e02d0a8ba4c30f6c756"},"cpuacct":{"id":"247dcdf397d729708a6c44e2113988babd67fdb774dc6e02d0a8ba4c30f6c756"},"memory":{"id":"247dcdf397d729708a6c44e2113988babd67fdb774dc6e02d0a8ba4c30f6c756"}},"cpu":{"system":{"ticks":130,"time":{"ms":133}},"total":{"ticks":290,"time":{"ms":297},"value":290},"user":{"ticks":160,"time":{"ms":164}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"43af8240-de5b-4074-b307-8d223898ade4","uptime":{"ms":30154}},"memstats":{"gc_next":16240032,"memory_alloc":14824312,"memory_total":44533136,"rss":67768320},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0.53,"15":0.34,"5":0.55,"norm":{"1":0.265,"15":0.17,"5":0.275}}}}}}