I was deployed ELK on local host and filebeat on remote host in order to fetch log from remote to local by using docker-compose for ELK and docker run for filebeat.
The flow will be like Filebeat(remote) -> Logstash (local) -> Elasticsearch(local) -> kibana(local)
The connection between Logstash and filebeat is great. Also the connection among ELK stack themselves.
At first, I've set multiple input paths in filebeat
- /var/log/*.log
- /var/log/message/.*log
- /deploy/tomcat/log/.*log
from these configuration, Kibana is able to discover a message but only for the first two paths. the /deploy/tomcat/log/.*log. So, I gave the permission (recursively) 777 to this path. Unfortunately, Filebeat harvester still doesn't work and keep saying
Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":860},"total":{"ticks":1550,"time":{"ms":4},"value":1550},"user":{"ticks":690,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"8877057c-bb0f-4363-b935-4db908c030e9","uptime":{"ms":4770025}},"memstats":{"gc_next":7600432,"memory_alloc":4448944,"memory_total":56104688},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.36,"15":0.46,"5":0.45,"norm":{"1":0.0225,"15":0.0288,"5":0.0281}}}}}}
Here's my filebeat configuration
filebeat.inputs:
- type: log
enabled: true
paths:
- "/deploy/tomcat/logs/*.log"
output.logstash:
hosts: ["http://x.x.x.x:5000"]
logging.level: debug
(my Logstash is running with port 5000)
What should I do next ? I've tried to change the configuration but result remain the same.