I am using Filebeat 6.3.0 to forward log files from a Rails application to Logstash. The files originate from a server that I cannot configure to use Filebeat, hence they are being transferred to the Filebeat/Logstash host via an rscript call.
The logfiles rotate on the server, so every once in a while, the existing log is rotated out to a
Basically, what I am doing is this:
rsync -avh user@remote:/path/to/logs/ /path/to/logs/
I have configured Filebeat to read the log files with the following input directives:
- type: log enabled: true paths: - /home/user/logs/production.log* fields: type: rails_production exclude_lines: ['^#', 'CSRF token authenticity'] - type: log enabled: true paths: - /home/user/logs/lograge_production.log* fields: type: rails_lograge exclude_lines: ['^#'] json.ignore_decoding_error: true json.keys_under_root: false json.add_error_key: false json.message_key: message
I have read this topic where it is suggested that if the files are being appended to, I should be fine. But as far as I can see,
rsync only transmits the delta anyway (as
--no-whole-file is the default for network transmissions).
Are my settings correct?
(The reason I am asking is that I am seeing input spikes in my timestamps, and other periods where there are no data being ingested for several hours, and this actually cannot be the case, as my data is coming in more or less constantly.)