# Logstash file output crashes when writing to append only filesystem

**URL:** <https://discuss.elastic.co/t/logstash-file-output-crashes-when-writing-to-append-only-filesystem/375569>\
**Category:** Logstash\
**Created:** [March 7, 2025, 5:16pm UTC](https://discuss.elastic.co/t/logstash-file-output-crashes-when-writing-to-append-only-filesystem/375569 "2025-03-07T17:16:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![hilsonp](https://avatars.discourse-cdn.com/v4/letter/h/f17d59/32.png) [@hilsonp](https://discuss.elastic.co/u/hilsonp)\
**Post date:** [March 7, 2025, 5:16pm UTC](https://discuss.elastic.co/t/logstash-file-output-crashes-when-writing-to-append-only-filesystem/375569/1 "2025-03-07T17:16:51Z")

</div>

Hello,

We have a logstash pipeline which output is configured to write to an append only nfs filesystem (WORM - NetApp SnapLock Volume Append Mode (VAM)).

File are being written but all of a sudden, the pipeline will crash with an exception showing a problem accessing the filesytem.  
I do copy the 2 kind of exceptions I see at the end of the post.

The logstash version is:

```auto
{"logstash.version"=>"8.4.2", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}

```

The pipeline output is:

```auto
    file {
      path => "/esb_worm/%{[legalLog][file][path]}"
      dir_mode => 0700
      file_mode => 0600
      flush_interval => 0
      write_behavior => "append"
    }

```

I already tried several flush\_interval values altought our final target is to set it to zero.

I did set the pipeline.workers to 1.

I tried different variations of nfs mount options.

Restarting logstash will let it run for a while then fail with one of the 2 errors provided here below and when a filename is given it will be a different one each time.

This seems the same problem than the one reported by @bjosve in this post: [Logstash file output crashes when writing to tamper proof storage](https://discuss.elastic.co/t/logstash-file-output-crashes-when-writing-to-tamper-proof-storage/172780)

What else can I try ? Is there anything I didn't try that is worth doing ?

> **Exception 1**
>
> [2025-03-07T17:09:36,823][ERROR][logstash.javapipeline][filestream-apigee] Pipeline worker error, the pipeline will be stopped {:pipeline\_id=\>"filestream-apigee", :error=\>"(SystemCallError) Unknown error (SystemCallError) - /esb\_worm/Legal-Traces/2025/02/04/dev/Legal-Traces-2025-02-04-dev-sd5325v.ndjson", :exception=\>Java::OrgJrubyExceptions::SystemCallError, :backtrace=\>["org.jruby.RubyIO.write(org/jruby/RubyIO.java:1504)", "org.jruby.RubyIO.write(org/jruby/RubyIO.java:1436)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.write(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:368)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.multi\_receive\_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.multi\_receive\_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1519)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.multi\_receive\_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:118)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:171)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.multi\_receive\_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:117)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.outputs.base.multi\_receive(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:103)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi\_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.java\_pipeline.start\_workers(/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:300)"], :thread=\>"#\<Thread:0xd207162 sleep\>"}

> **Exception 2**
>
> [2025-03-07T17:14:02,833][ERROR][logstash.javapipeline][filestream-apigee] Pipeline worker error, the pipeline will be stopped {:pipeline\_id=\>"filestream-apigee", :error=\>"(SystemCallError) Unknown error (SystemCallError) - /worm/2025/02/04/dev/2025-02-04-dev.ndjson", :exception=\>Java::OrgJrubyExceptions::SystemCallError, :backtrace=\>["org.jruby.RubyIO.write(org/jruby/RubyIO.java:1504)", "org.jruby.RubyIO.write(org/jruby/RubyIO.java:1436)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.write(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:368)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.multi\_receive\_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.multi\_receive\_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1519)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.multi\_receive\_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:118)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:171)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_output\_minus\_file\_minus\_4\_dot\_3\_dot\_0.lib.logstash.outputs.file.multi\_receive\_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:117)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.outputs.base.multi\_receive(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:103)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi\_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.java\_pipeline.start\_workers(/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:300)"], :thread=\>"#\<Thread:0x401d5539@/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:52 sleep\>"}

I also found those (maybe) related topics:

> <https://github.com/logstash-plugins/logstash-output-file/issues/51>
>
> We are testing the ability for logstash to write to an append only file for secu…rity purposes. (Ubuntu 14.04/Logstash 5.2.2)
> 
> We let logstash create and open the file, then set the target file to append only, using \`chattr +a {file}\`. 
> 
> Logstash is able to continue writing to the file but once it closes and then attempts to open the file again logstash produces this error and crashes: 
> https://gist.github.com/erikanderson/fd243942d7119bb116baec2318ff6b6e
> \`\`\`
> \[2017-03-13T14:34:24,744\]\[FATAL\]\[logstash.runner \] An unexpected error occurred! {:error=\>#\<Errno::EACCES: Permission denied - /var/local/logstash/\*\*\*\*-2017.03.13\>, :backtrace=\>\["org/jruby/RubyFile.java:370:in \`initialize'", "org/jruby/RubyIO.java:871:in \`new'", "/srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.1/lib/logstash/outputs/file.rb:280:in \`open'", "/srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.1/lib/logstash/outputs/file.rb:132:in \`multi\_receive\_encoded'", "org/jruby/RubyHash.java:1342:in \`each'", "/srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.1/lib/logstash/outputs/file.rb:131:in \`multi\_receive\_encoded'", "org/jruby/ext/thread/Mutex.java:149:in \`synchronize'", "/srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.1/lib/logstash/outputs/file.rb:130:in \`multi\_receive\_encoded'", "/srv/logstash/logstash-5.2.2/logstash-core/lib/logstash/outputs/base.rb:90:in \`multi\_receive'", "/srv/logstash/logstash-5.2.2/logstash-core/lib/logstash/output\_delegator\_strategies/shared.rb:12:in \`multi\_receive'", "/srv/logstash/logstash-5.2.2/logstash-core/lib/logstash/output\_delegator.rb:43:in \`multi\_receive'", "/srv/logstash/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:414:in \`output\_batch'", "org/jruby/RubyHash.java:1342:in \`each'", "/srv/logstash/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:413:in \`output\_batch'", "/srv/logstash/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:371:in \`worker\_loop'", "/srv/logstash/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:331:in \`start\_workers'"\]}
> \[2017-03-13T14:34:24,756\]\[DEBUG\]\[logstash.agent \] Error in reactor loop escaped: Bad file descriptor - Bad file descriptor (Errno::EBADF)
> \[2017-03-13T14:34:24,757\]\[DEBUG\]\[logstash.agent \] \["org/jruby/RubyIO.java:3705:in \`select'", "/srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/reactor.rb:29:in \`run\_internal'", "/srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/reactor.rb:138:in \`run\_in\_thread'"\]
> \[2017-03-13T14:34:24,757\]\[DEBUG\]\[logstash.agent \] 2017-03-13 14:34:24 -0600: Listen loop error: #\<Errno::EBADF: Bad file descriptor - Bad file descriptor\>
> \[2017-03-13T14:34:24,757\]\[DEBUG\]\[logstash.agent \] org/jruby/RubyIO.java:3705:in \`select'
> /srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:322:in \`handle\_servers'
> /srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:296:in \`run'
> \[2017-03-13T14:34:24,757\]\[DEBUG\]\[logstash.agent \] Error in reactor loop escaped: Bad file descriptor - Bad file descriptor (Errno::EBADF)
> \[2017-03-13T14:34:24,757\]\[DEBUG\]\[logstash.agent \] \["org/jruby/RubyIO.java:3705:in \`select'", "/srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/reactor.rb:29:in \`run\_internal'", "/srv/logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/reactor.rb:138:in \`run\_in\_thread'"\]
> \`\`\`
> I realize this is probably an issue within ruby/jruby and I am going to research if there is a way to accomodate this.

> [@File output with append only files](https://discuss.elastic.co/t/file-output-with-append-only-files/78064):
>
> We are experimenting with setting up a file output to store certain types of logs. Right after the file is created by logstash we are using incron to set the file to append only with chattr +a. Logstash writes to the file fine after this change. After a period of inactivity logstash closes the file. When it goes to reopen the file it causes a crashing error that it can't access the file. I'm not seeing any explicit option in the documentation but Is there a way to disable logstash closing the …

Thanks in advance,

Pierre