FileBeat has been running as a docker container started by docker-compose file and already mapped the volume for filebeat.yml.
I've updated the configuration in filebeat.yml by setting input and output as Elasticsearch.
filebeat.inputs:
- type: log
enabled: true
paths:
# - "/usr/share/filebeat/logs/*.log"
#- "/deploy/tomcat/logs/*.access*.log"
output.elasticsearch:
hosts: ["deploy-01.com:9200"]
username: "elastic"
password: "changeme"
indices:
- index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"
- index: "warning-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
message: "WARN"
- index: "error-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
message: "ERR"
setup.kibana:
host: "deploy-01.com:5601"
username: "elastic"
password: "changeme"
It turns out that the FileBeat only harvest once for the first time after the fresh installation. Although the logs in the provided directory always create every day.
Checked for the permission of the directory and files
docker logs <filebeat_container> only gives these
2020-09-30T10:29:24.087Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":360},"total":{"ticks":1560,"time":{"ms":7},"value":1560},"user":{"ticks":1200,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"c038389c-cdac-4f29-bff1-69e94bda3d36","uptime":{"ms":3810205}},"memstats":{"gc_next":8119184,"memory_alloc":4486864,"memory_total":50182328},"runtime":{"goroutines":20}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":7.27,"15":7.6,"5":7.56,"norm":{"1":0.2272,"15":0.2375,"5":0.2363}}}}}}
Restart the container, re-create index pattern doesn't help.
Here are the logs in the directory, created every day
-rw-rw-r-- 1 deploy deploy 271 Sep 27 15:30 deploy.com.access.2020-09-27.log
-rw-rw-r-- 1 deploy deploy 70876 Sep 28 18:28 deploy.com.access.2020-09-28.log
-rw-rw-r-- 1 deploy deploy 295 Sep 29 04:20 deploy.com.access.2020-09-29.log
-rw-rw-r-- 1 deploy deploy 3268 Sep 30 11:05 deploy.com.access.2020-09-30.log
Kibana dashboard doesn't show any new logs
Please help,
Thanks