Question about logrotate with gzip

Elastic Stack Beats
1

Hi,

I have a question about the behavior of filebeat with logrotate and gzip.

I had a problem with my logstash server that takes all filebeats events. This logstash was stopped so all filebeats could send logs to it.

These files have been gzipped from logrotate, then I restarted the server and I saw gzipped logs in Elasticsearch. I was surprised.

Can you explain how can filebeat manage these cases?

It's for this?

Each prospector keeps a state for each file it finds. Because files can be renamed or moved, the filename and path are not enough to identify a file. For each file, Filebeat stores unique identifiers to detect whether a file was harvested previously.

Thank you so much

2

Filebeat keeps files open until the data is shipped. So I think what happened in your case is that the files were gzip but the unzip files were still around. So when Logstash became available again, the still open files finished shipping. Filebeat currently does not support gzip files, but there is some work done on it: https://github.com/elastic/beats/pull/3070

3

This topic was automatically closed after 21 days. New replies are no longer allowed.