I have configured filebeat(version 7.1.1) on my system to send logs to AWS Elastic Search but it is not doing following two things appropriately:
- Updating it's own log file at
/var/log/filebeat, and - Sending the updated logs from
/var/log/*logsto AWS ES
Issue 1: When I run sudo service filebeat restart it does not update logs files. I tried running filebeat -c /etc/filebeat/filebeat.yml to update the log files. When I run this command the terminal hangs and I have to manually close the terminal and start a new session. Filebeat logs get updated after this.
Issue 2: The log files are sent only the first time and then it never gets updated. When I do curl XGET aws_elasticsearch_domain it only returns the logs sent for the first time.I checked the Kibana dashboard as well and it only has the logs from the first time.
As per my current /var/log/filebeat/filebeat log file the connection to ES was successful and the logs were send but I still cannot see the updated logs on Kibana or through curl -XGET vpc-xxxxxxx.es.amazonaws.com/_search?pretty=true&q=*:*.
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
scan_frequency: 5s
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
setup.template.settings:
index.number_of_shards: 1
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.elasticsearch:
hosts: ["vpc-xxxxxxxxxx.es.amazonaws.com:80"]
bulk_max_size: 2
timeout: 180
setup.kibana:
host: ["vpc-xxxxxxxxxxx.es.amazonaws.com:80"]
path: /_plugin/kibana