I am running Filebeat inside docker container. My Filebeat service is working correctly on mac os and ubuntu machine but on another ubuntu vm it's not working. Harvesters are not started for the files neither connection to the Kafka is established. Can anyone guide me in this regard.
The code of my filebeat.yml is given below:
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
templates:
- condition.or:
- contains:
docker.container.image: logstash
- contains:
docker.container.image: elasticsearch
- contains:
docker.container.image: zookeeper
- contains:
docker.container.image: kafka
config:
- type: container
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
multiline.type: pattern
multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
hints.default_config:
- type: container
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
processors:
- add_docker_metadata:
host: "unix:///var/run/docker.sock"
# Output
output.kafka:
hosts: '${KAFKA_HOSTS:kafka:9092}'
topic: "topicname"
The entries in my logs are like this:
filebeat_1 | 2021-04-19T22:01:59.554Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/0d6c1cb66f536bc4eb0f54b79bdf724efd7d9b924f95852587259c4e8e436cf7/.log]
filebeat_1 | 2021-04-19T22:01:59.556Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/0d6c1cb66f536bc4eb0f54b79bdf724efd7d9b924f95852587259c4e8e436cf7/.log]
filebeat_1 | 2021-04-19T22:01:59.557Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/02a8d44666b3d23c40634abd810624eda6a37d916c2bd3ae2af385809a3e0949/*-json.log]
filebeat_1 | 2021-04-19T22:02:29.386Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":{"ms":117}},"total":{"ticks":530,"time":{"ms":544},"value":530},"user":{"ticks":420,"time":{"ms":427}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":13},"info":{"ephemeral_id":"67603890-9854-43cb-8acc-24f0f1b63940","uptime":{"ms":30324}},"memstats":{"gc_next":16842160,"memory_alloc":10584560,"memory_total":42847104,"rss":66187264},"runtime":{"goroutines":324}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":26,"starts":26}},"output":{"type":"kafka"},"pipeline":{"clients":26,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4},"load":{"1":2.51,"15":1.9,"5":1.81,"norm":{"1":0.6275,"15":0.475,"5":0.4525}}}}}}
filebeat_1 | 2021-04-19T22:02:59.385Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":130,"time":{"ms":19}},"total":{"ticks":570,"time":{"ms":32},"value":570},"user":{"ticks":440,"time":{"ms":13}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":13},"info":{"ephemeral_id":"67603890-9854-43cb-8acc-24f0f1b63940","uptime":{"ms":60322}},"memstats":{"gc_next":16842160,"memory_alloc":10852768,"memory_total":43115312},"runtime":{"goroutines":324}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":26}},"pipeline":{"clients":26,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":2.26,"15":1.9,"5":1.82,"norm":{"1":0.565,"15":0.475,"5":0.455}}}}}}
filebeat_1 | 2021-04-19T22:11:59.471Z INFO [docker.docker] docker/watcher.go:328 No events received within 10m0s, restarting watch call
filebeat_1 | 2021-04-19T22:12:00.479Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/02a8d44666b3d23c40634abd810624eda6a37d916c2bd3ae2af385809a3e0949/*-json.log]
From the logs, I can clearly see that harvesters are not started. I am unable to figure out why this is only happening in this ubuntu vm not on the other machine. Can anyone guide me please?