S3 output can't find files it generates

Elastic Stack Logstash
1

Hi, I'm using Logstash 5.0.2 (and .0.1 before) and I need to add an s3 output but I can't because randomly one of the file logstash generates cannot be found:

[2016-12-07T14:37:39,809][ERROR][logstash.outputs.s3      ] failed to upload, will re-enqueue /tmp/logstash/ls.s3.ip-172-31-45-248.2016-12-07T14.31.part0.txt for upload {:ex=>#<Errno::ENOENT: No such file or directory - /tmp/logstash/ls.s3.ip-172-31-45-248.2016-12-07T14.31.part0.txt>, :backtrace=>["org/jruby/RubyFile.java:370:in `initialize'", "org/jruby/RubyIO.java:1197:in `open'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-3.2.0/lib/logstash/outputs/s3.rb:175:in `write_on_bucket'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-3.2.0/lib/logstash/outputs/s3.rb:289:in `move_file_to_bucket'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-3.2.0/lib/logstash/outputs/s3.rb:454:in `upload_worker'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-3.2.0/lib/logstash/outputs/s3.rb:436:in `configure_upload_workers'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in `initialize'"]}

As it says Logstash "will re-enqueue" the file over and over again filling up the disk with log messages. This happens randomly: Logstash works for some time (minutes or hours) and then suddenly this happens.

This is my output configuration:

if [type] == "my-log" {
  s3 {
    bucket => "mybucket"
    prefix => "logstash/bla/"
    size_file => 10485760  # rotate every 10M
    codec => line {
      format => "%{message}"
    }
  }
}

I'm on ubuntu 14.04. Oracle JVM:

# java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

Hints?

2

Happening to me as well on 5.1.1.

Fills up up 20gb in less than an hour, about to have to disable it.

Also note this in a 3 node logstash cluster.

3

Fixed mine by removing the size_file / time_file options.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.