Logstash file output crashes when writing to append only filesystem

[hilsonp (Pierre Hilson)]

Hello,

We have a logstash pipeline which output is configured to write to an append only nfs filesystem (WORM - NetApp SnapLock Volume Append Mode (VAM)).

File are being written but all of a sudden, the pipeline will crash with an exception showing a problem accessing the filesytem.
I do copy the 2 kind of exceptions I see at the end of the post.

The logstash version is:

{"logstash.version"=>"8.4.2", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}

The pipeline output is:

    file {
      path => "/esb_worm/%{[legalLog][file][path]}"
      dir_mode => 0700
      file_mode => 0600
      flush_interval => 0
      write_behavior => "append"
    }

I already tried several flush_interval values altought our final target is to set it to zero.

I did set the pipeline.workers to 1.

I tried different variations of nfs mount options.

Restarting logstash will let it run for a while then fail with one of the 2 errors provided here below and when a filename is given it will be a different one each time.

This seems the same problem than the one reported by @bjosve in this post: Logstash file output crashes when writing to tamper proof storage

What else can I try ? Is there anything I didn't try that is worth doing ?

Exception 1

[2025-03-07T17:09:36,823][ERROR][logstash.javapipeline ][filestream-apigee] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"filestream-apigee", :error=>"(SystemCallError) Unknown error (SystemCallError) - /esb_worm/Legal-Traces/2025/02/04/dev/Legal-Traces-2025-02-04-dev-sd5325v.ndjson", :exception=>Java::OrgJrubyExceptions::SystemCallError, :backtrace=>["org.jruby.RubyIO.write(org/jruby/RubyIO.java:1504)", "org.jruby.RubyIO.write(org/jruby/RubyIO.java:1436)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.write(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:368)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1519)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:118)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:171)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:117)", "usr.share.logstash.logstash_minus_core.lib.logstash.outputs.base.multi_receive(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:103)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300)"], :thread=>"#<Thread:0xd207162 sleep>"}

Exception 2

[2025-03-07T17:14:02,833][ERROR][logstash.javapipeline ][filestream-apigee] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"filestream-apigee", :error=>"(SystemCallError) Unknown error (SystemCallError) - /worm/2025/02/04/dev/2025-02-04-dev.ndjson", :exception=>Java::OrgJrubyExceptions::SystemCallError, :backtrace=>["org.jruby.RubyIO.write(org/jruby/RubyIO.java:1504)", "org.jruby.RubyIO.write(org/jruby/RubyIO.java:1436)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.write(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:368)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1519)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:118)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:171)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:117)", "usr.share.logstash.logstash_minus_core.lib.logstash.outputs.base.multi_receive(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:103)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300)"], :thread=>"#<Thread:0x401d5539@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52 sleep>"}

I also found those (maybe) related topics:

Unable to write to file after it is set to append only and closed/opened

File output with append only files

We are experimenting with setting up a file output to store certain types of logs. Right after the file is created by logstash we are using incron to set the file to append only with chattr +a. Logstash writes to the file fine after this change. After a period of inactivity logstash closes the file. When it goes to reopen the file it causes a crashing error that it can't access the file. I'm not seeing any explicit option in the documentation but Is there a way to disable logstash closing the …

Thanks in advance,

Pierre