Have a nice day everyone.
Not long ago, I started to notice missing syslog logs from random host groups.
At first, I simply restarted our Logstash nodes that act as syslog forwarders, and the problem went away.
But over the last two days, I’ve seen this happen twice, so I had to start digging deeper.
At the beginning, I suspected some network-related issues, but then I correlated the last host message with several errors on a particular Logstash node.
In most cases, the problem starts with one or more errors that look like this:
[2025-11-18T01:54:17,368][ERROR][logstash.outputs.file ][pipeline_name] Exception flushing files {:exception=>"Unknown error (SystemCallError) - ", :backtrace=>["org/jruby/RubyIO.java:2501:in `flush'", "C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:373:in `flush'", "C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:208:in `block in flush_pending_files'", "org/jruby/RubyHash.java:1615:in `each'", "C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:206:in `block in flush_pending_files'", "org/jruby/ext/thread/Mutex.java:174:in `synchronize'", "C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:203:in `flush_pending_files'", "C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:103:in `block in register'", "C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:350:in `block in run'", "org/jruby/RubyKernel.java:1725:in `loop'", "C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:346:in `block in run'", "org/jruby/ext/thread/Mutex.java:174:in `synchronize'", "C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:345:in `run'", "C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:321:in `block in start'"]}
Then I see this error and end up with a corrupted pipeline, and I have to restart the Logstash node manually:
[2025-11-18T01:55:53,562][ERROR][logstash.javapipeline ][pipeline_name] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"pipeline_name", :error=>"(SystemCallError) Unknown error (SystemCallError) - E:/logstash/group_name/pipeline_name/hostname/2025-11-17-22.log", :exception=>Java::OrgJrubyExceptions::SystemCallError, :backtrace=>["org.jruby.RubyIO.write(org/jruby/RubyIO.java:1590)", "org.jruby.RubyIO.write(org/jruby/RubyIO.java:1522)", "C_3a_.Program_20_Files.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.write(C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:368)", "C_3a_.Program_20_Files.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1981)", "C_3a_.Program_20_Files.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1615)", "C_3a_.Program_20_Files.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:118)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:174)", "C_3a_.Program_20_Files.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(C:/Program Files/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:117)", "C_3a_.Program_20_Files.Logstash.logstash_minus_core.lib.logstash.outputs.base.multi_receive(C:/Program Files/Logstash/logstash-core/lib/logstash/outputs/base.rb:102)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "RUBY.start_workers(C:/Program Files/Logstash/logstash-core/lib/logstash/java_pipeline.rb:308)"], :thread=>"#<Thread:0x7bdaebf8 C:/Program Files/Logstash/logstash-core/lib/logstash/java_pipeline.rb:138 sleep>"}
Here some information related to my installation.
Logstash Version: 9.0.0
OS: Windows Server 2019
Pipeline-related config:
>pipelines.yml
- pipeline.id: pipeline_name
path.config: "C:/Program Files/logstash/etc/pipelines/pipeline_name.conf"
>pipeline_name.conf
input {
udp {
port => 1118
}
}
filter {
dns {
action => "replace"
hit_cache_size => 500000
hit_cache_ttl => 300
max_retries => 10
timeout => 3
reverse => [ "[host]" ]
}
}
output {
file {
codec => line { format => "%{message}" }
path => "E:/logstash/host_group/pipeline_name/%{[host]}/%{+YYYY-MM-dd-HH}.log"
}
}