Getting ENOSPC error when using file output plugin in Logstash

Elastic Stack Logstash
1

Hi Team,
I have the following logstash pipeline config.

input {
    tcp {
        port => "${TCP_PORT}"
        codec => line
    }
}
filter {
    grok {
        match => {"message" => "%{SYSLOGTIMESTAMP:time} %{DATA:stream_id} %{DATA:trace_name} %{DATA:node_name} %{DATA:count} %{DATA:thread_id} %{GREEDYDATA:data}"}
        match => {"message" => "%{SYSLOGTIMESTAMP:time} %{DATA:stream_id} %{DATA:trace_name} %{DATA:node_name} %{DATA:thread_id} %{GREEDYDATA:data}"}
    }
    mutate {
      gsub => [
        # replace all forward slashes with underscore
        "node_name", "/", "_"
      ]
    }
}
output {
    file {
        path => "/usr/share/logstash/data/%{stream_id}/%{node_name}/%{trace_name}.txt"
        codec => line
    }
    stdout {
        codec => rubydebug
    }
}

I am starting logstash as a docker container and getting below error.

[2023-05-08T05:01:26,115][INFO ][logstash.outputs.file    ][main][34d6886ed0ddbf26407ed825a2eea273f7e9039267356b9bed0f5f2c05b422c7] Closing file /usr/share/logstash/data/tb5-r1/tb5-r1/0_RP0_CPU0/isis/1/spf.txt
[2023-05-08T05:01:27,449][INFO ][logstash.outputs.file    ][main][34d6886ed0ddbf26407ed825a2eea273f7e9039267356b9bed0f5f2c05b422c7] Opening file {:path=>"/usr/share/logstash/data/tb5-r1/tb5-r1/0_RP0_CPU0/isis/2/std.txt"}
[2023-05-08T05:01:36,116][INFO ][logstash.outputs.file    ][main][34d6886ed0ddbf26407ed825a2eea273f7e9039267356b9bed0f5f2c05b422c7] Closing file /usr/share/logstash/data/tb5-r1/tb5-r1/0_RP0_CPU0/isis/1/sev.txt
[2023-05-08T05:01:45,923][ERROR][logstash.javapipeline    ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"(ENOSPC) No space left on device - /usr/share/logstash/data/tb5-r1/tb5-r1/0_RP0_CPU0/isis/2/upd.txt", :exception=>Java::OrgJrubyExceptions::SystemCallError, :backtrace=>["org.jruby.RubyIO.write(org/jruby/RubyIO.java:1504)", "org.jruby.RubyIO.write(org/jruby/RubyIO.java:1436)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.write(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:368)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:126)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1519)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:118)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:171)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:117)", "usr.share.logstash.logstash_minus_core.lib.logstash.outputs.base.multi_receive(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:103)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "RUBY.start_workers(/usr/share/logstash/logs

I verified that there is enough disk space available (approx 7GB ) to write the file. Is there some limit on the maximum file size that the file output plugin can write. The "ls" output shows current file size on the disk:

next@next-exec-server-1:~/logstash$ ls -l logs/tb5-r1/tb5-r1/0_RP0_CPU0/isis/2/
total 921740
-rw-r--r-- 1 next next 143818730 May  7 22:00 spf.txt
-rw-r--r-- 1 next next 148688348 May  7 22:01 std.txt
-rw-r--r-- 1 next next   2010785 May  7 21:57 te_.txt
-rw-r--r-- 1 next next 649330684 May  7 22:25 upd.txt

Thanks,
Arinjay

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.