Hi everyone,
we are using the file output plugin with gzip=true and we are getting quite a bit of corrupted files. At first we thought that it had to do with the size of the files but it appears quite randomly. Some of our files are +2GB when compressed. I looked into the file output plugin code and noticed that when new files are crated they are actually opened with
File.new(path, "a+")
https://github.com/logstash-plugins/logstash-output-file/blob/master/lib/logstash/outputs/file.rb line 262 and 264 and I am now wondering if this should be
File.new(path, "ab+")
in the case of gzip files being binary. I am unfortunately not familiar with jRuby and am not sure if this creates the problem. Any help is highly appreciated.
Best
Ingo