Hi,
I recently migrated to logstash/logstash:6.8.23 docker image, to benefit from the archive checking setting...
I see repeated logs about corrupted archives, but cannot figure out if they apply to the same filepath (seeing the filepath would be nice, if I may suggest).
How does logstash behave when encountering a corrupted archive: does it mark it and eventually skips it ? Or will it eventually retry the file ?
Thanks for your help,
Olivier
--- related logs from logstash:
logstash_fdmch_history_viewer | Pipeline_id:prod_fdm_history
logstash_fdmch_history_viewer | Plugin: <LogStash::Inputs::File start_position=>"beginning", check_archive_validity=>true, sincedb_clean_afte r=>7200.0, file_chunk_count=>2, file_sort_by=>"path", file_chunk_size=>1048576, file_sort_direction=>"asc", file_completed_action=>"log", file_ completed_log_path=>"/dev/null", close_older=>120.0, max_open_files=>1024, mode=>"read", path=>["/data_nfs/fdm/LOGS/production/2022/05/21/FLM-F LIGHT/**/FDM_history_*.log.gz", "/data_nfs/fdm/LOGS/production/2022/05/22/FLM-FLIGHT/**/FDM_history_*.log.gz", "/data_nfs/fdm/LOGS/production/2 022/05/23/FLM-FLIGHT/**/FDM_history_*.log.gz", "/data_nfs/fdm/LOGS/production/2022/05/24/FLM-FLIGHT/**/FDM_history_*.log.gz", "/data_nfs/fdm/LO GS/production/2022/05/25/FLM-FLIGHT/**/FDM_history_*.log.gz", "/data_nfs/fdm/LOGS/production/2022/05/26/FLM-FLIGHT/**/FDM_history_*.log.gz", "/ data_nfs/fdm/LOGS/production/2022/05/27/FLM-FLIGHT/**/FDM_history_*.log.gz", "/data_nfs/fdm/LOGS/production/2022/05/28/FLM-FLIGHT/**/FDM_histor y_*.log.gz", "/data_nfs/fdm/LOGS/production/2022/05/29/FLM-FLIGHT/**/FDM_history_*.log.gz", "/data_nfs/fdm/LOGS/production/2022/05/30/FLM-FLIGH T/**/FDM_history_*.log.gz", "/data_nfs/fdm/LOGS/production/2022/05/31/FLM-FLIGHT/**/FDM_history_*.log.gz"], codec=><LogStash::Codecs::Multiline pattern=>"^%{SPACE}%{TIMESTAMP_ISO8601}", what=>"previous", id=>"c7e7c0db-aadc-4c3c-a6b8-5acbf59ec5f3", max_lines=>2000, negate=>true, enable_ metric=>true, charset=>"UTF-8", multiline_tag=>"multiline", max_bytes=>10485760>, delimiter=>"\\n", id=>"production-files-2022-05_FLM_FLIGHT", sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/production/fdm_history_FLM_SERVICE_2022-05.sincedb", enable_metric=>true, stat_inte rval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, exit_after_read=>false>
logstash_fdmch_history_viewer | Error: Unexpected end of ZLIB input stream
logstash_fdmch_history_viewer | Exception: Java::JavaIo::EOFException
logstash_fdmch_history_viewer | Stack: java.util.zip.InflaterInputStream.fill(java/util/zip/InflaterInputStream.java:240)
logstash_fdmch_history_viewer | java.util.zip.InflaterInputStream.read(java/util/zip/InflaterInputStream.java:158)
logstash_fdmch_history_viewer | java.util.zip.GZIPInputStream.read(java/util/zip/GZIPInputStream.java:117)
logstash_fdmch_history_viewer | java.io.FilterInputStream.read(java/io/FilterInputStream.java:107)
logstash_fdmch_history_viewer | java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
logstash_fdmch_history_viewer | org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:455)
logstash_fdmch_history_viewer | org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:316)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.handlers.read_zip_file.invokeOther8:read(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_inp ut_minus_file_minus_4_dot_1_dot_18/lib/filewatch/read_mode/handlers//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1. 18/lib/filewatch/read_mode/handlers/read_zip_file.rb:76)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.handlers.read_zip_file.corrupted?(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/fi lewatch/read_mode/handlers/read_zip_file.rb:76)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.handlers.read_zip_file.invokeOther9:corrupted?(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_min us_input_minus_file_minus_4_dot_1_dot_18/lib/filewatch/read_mode/handlers//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-fil e-4.1.18/lib/filewatch/read_mode/handlers/read_zip_file.rb:22)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.handlers.read_zip_file.handle_specifically(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1. 18/lib/filewatch/read_mode/handlers/read_zip_file.rb:22)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.handlers.base.invokeOther8:handle_specifically(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_min us_input_minus_file_minus_4_dot_1_dot_18/lib/filewatch/read_mode/handlers//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-fil e-4.1.18/lib/filewatch/read_mode/handlers/base.rb:26)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.handlers.base.handle(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/read_ mode/handlers/base.rb:26)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.processor.invokeOther1:handle(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_minus_fi le_minus_4_dot_1_dot_18/lib/filewatch/read_mode//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/rea d_mode/processor.rb:39)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.processor.read_zip_file(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/re ad_mode/processor.rb:39)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.processor.invokeOther5:read_zip_file(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_m inus_file_minus_4_dot_1_dot_18/lib/filewatch/read_mode//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewa tch/read_mode/processor.rb:102)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.read_mode.processor.process_active(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/r ead_mode/processor.rb:102)
logstash_fdmch_history_viewer | org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792)
logstash_fdmch_history_viewer | org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)
logstash_fdmch_history_viewer | RUBY.process_active(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch /read_mode/processor.rb:88)
logstash_fdmch_history_viewer | RUBY.process_all_states(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filew atch/read_mode/processor.rb:45)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.watch.invokeOther5:process_all_states(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_minus_file _minus_4_dot_1_dot_18/lib/filewatch//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/watch.rb:68)
logstash_fdmch_history_viewer | usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_18 .lib.filewatch.watch.iterate_on_state(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/watch.rb:68)
logstash_fdmch_history_viewer | RUBY.subscribe(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/watc h.rb:45)
logstash_fdmch_history_viewer | RUBY.subscribe(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/obse rving_read.rb:12)
logstash_fdmch_history_viewer | RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/logstash/inputs/file .rb:363)
logstash_fdmch_history_viewer | RUBY.inputworker(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426)
logstash_fdmch_history_viewer | RUBY.start_input(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420)
logstash_fdmch_history_viewer | org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)
logstash_fdmch_history_viewer | org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)
logstash_fdmch_history_viewer | org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)
logstash_fdmch_history_viewer | java.lang.Thread.run(java/lang/Thread.java:748)
logstash_fdmch_history_viewer | [2022-05-30T08:40:01,007][INFO ][filewatch.observingread ] QUIT - closing all files and shutting down.
logstash_fdmch_history_viewer | [2022-05-30T08:40:01,007][INFO ][filewatch.observingread ] START, creating Discoverer, Watch with file and sin cedb collections