Error in logstash-input-s3-sns-sqs plugin

Elastic Stack Logstash
1 
Hi all,
          I am new to logstash. I am using s3snssqs plugin to pull data from s3.
 
input {

  s3snssqs {

    region                     => "us-east-1"

    s3_default_options         => { "endpoint_discovery" => true }

    queue                      => "testingjkqueue"

    queue_owner_aws_account_id => "1111111111111"

    access_key_id              => "***"

    secret_access_key          => "*****" 

    sqs_skip_delete            => true

    codec                      => json

  }

}

output {

    stdout { codec => rubydebug }

}

This is my conf file. I am getting the below error when I execute this conf file.


SystemCallError: Unknown error (SystemCallError) - Unknown Error (20109) - C:\Users\user\AppData\Local\Temp\C:UsersuserAppDataLocalTemplogstash20200812-8076-3bbqp7
              mkdir at org/jruby/RubyDir.java:632
           mktmpdir at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:89
             create at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:137
           mktmpdir at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:87
  run_worker_thread at D:/logstash/logstash-7.8.0-new/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:333
warning: thread "Ruby-0-Thread-14: :1" terminated with exception (report_on_exception is true):
SystemCallError: Unknown error (SystemCallError) - Unknown Error (20109) - C:\Users\user\AppData\Local\Temp\C:UsersuserAppDataLocalTemplogstash20200812-8076-3bbqp7
              mkdir at org/jruby/RubyDir.java:632
           mktmpdir at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:89
             create at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:137
           mktmpdir at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:87
  run_worker_thread at D:/logstash/logstash-7.8.0-new/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:333
[2020-08-12T15:51:25,277][ERROR][logstash.javapipeline    ][main][44c1515dfee0dc84029c82f232b5448867cb480791b7f65102c4dc8c8e8f04dc] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::S3SNSSQS access_key_id=>"*******", codec=><LogStash::Codecs::JSON id=>"json_d6247713-b307-4bda-b4b3-abdd1dc3d91c", enable_metric=>true, charset=>"UTF-8">, s3_default_options=>{"endpoint_discovery"=>"true"}, queue_owner_aws_account_id=>"11111111111", secret_access_key=><password>, sqs_skip_delete=>true, id=>"44c1515dfee0dc84029c82f232b5448867cb480791b7f65102c4dc8c8e8f04dc", region=>"us-east-1", queue=>"testingjkqueue", enable_metric=>true, threads=>1, role_session_name=>"logstash", s3_role_session_name=>"logstash", delete_on_success=>false, include_object_properties=>[:last_modified, :content_type, :metadata], from_sns=>true, sqs_delete_on_failure=>true, visibility_timeout=>120, max_processing_time=>8000, temporary_directory=>"C:/Users/user/AppData/Local/Temp/logstash", consumer_threads=>1>
  Error: Unknown error (SystemCallError) - Unknown Error (20109) - C:\Users\user\AppData\Local\Temp\C:UsersuserAppDataLocalTemplogstash20200812-8076-3bbqp7
  Exception: SystemCallError
  Stack: org/jruby/RubyDir.java:632:in `mkdir'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:89:in `block in mktmpdir'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:137:in `create'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:87:in `mktmpdir'
D:/logstash/logstash-7.8.0-new/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:333:in `block in run_worker_thread'
[2020-08-12T15:51:25,359][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<SystemCallError: Unknown error (SystemCallError) - Unknown Error (20109) - C:\Users\user\AppData\Local\Temp\C:UsersuserAppDataLocalTemplogstash20200812-8076-3bbqp7>, :backtrace=>["org/jruby/RubyDir.java:632:in `mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:89:in `block in mktmpdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:137:in `create'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:87:in `mktmpdir'", "D:/logstash/logstash-7.8.0-new/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:333:in `block in run_worker_thread'"]}
[2020-08-12T15:51:25,384][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Could someone help me to resolve this issue?

Thanks.
2

It is trying to create a temporary directory to work in, and clearly that is not a valid directory name. I do not understand how the default value could result in that, but I suggest explicitly setting the temporary_directory option on the input.

3 
I have created a temporary directory but I am getting the same error.

input {
  s3snssqs {
    region                     => "us-east-1"
    s3_default_options         => { "endpoint_discovery" => true }
    queue                      => "testingqueue"
    queue_owner_aws_account_id => "******"
    access_key_id              => "*********"
    secret_access_key          => "***********" 
    sqs_skip_delete            => true
    codec                      => json
    from_sns                   => false
    temporary_directory        => "D:\Temp"
    s3_options_by_bucket => [
                                {
                                    bucket_name => "testingbucket"
                                }
                            ]

  }                 
}

output {
    stdout { codec => rubydebug }
}


Error:

SystemCallError: Unknown error (SystemCallError) - Unknown Error (20109) - C:\Users\user\AppData\Local\Temp\D:Temp20200812-10396-1vyo3pr
              mkdir at org/jruby/RubyDir.java:632
           mktmpdir at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:89
             create at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:137
           mktmpdir at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:87
  run_worker_thread at D:/logstash/logstash-7.8.0-new/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:333
warning: thread "Ruby-0-Thread-13: :1" terminated with exception (report_on_exception is true):
SystemCallError: Unknown error (SystemCallError) - Unknown Error (20109) - C:\Users\user\AppData\Local\Temp\D:Temp20200812-10396-1vyo3pr
              mkdir at org/jruby/RubyDir.java:632
           mktmpdir at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:89
             create at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:137
           mktmpdir at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:87
  run_worker_thread at D:/logstash/logstash-7.8.0-new/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:333
[2020-08-12T20:52:32,857][ERROR][logstash.javapipeline    ][main][046de1113aee8af58dc67ef6579674e7f50f2e20a017effdedc3410364361042] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::S3SNSSQS access_key_id=>"***", codec=><LogStash::Codecs::JSON id=>"json_05e47943-d45c-4380-a2b8-b7944bae8c09", enable_metric=>true, charset=>"UTF-8">, s3_options_by_bucket=>[{"bucket_name"=>"testingbucket"}], s3_default_options=>{"endpoint_discovery"=>"true"}, from_sns=>false, queue_owner_aws_account_id=>"******", temporary_directory=>"D:\\Temp", secret_access_key=><password>, sqs_skip_delete=>true, id=>"046de1113aee8af58dc67ef6579674e7f50f2e20a017effdedc3410364361042", region=>"us-east-1", queue=>"testingqueue", enable_metric=>true, threads=>1, role_session_name=>"logstash", s3_role_session_name=>"logstash", delete_on_success=>false, include_object_properties=>[:last_modified, :content_type, :metadata], sqs_delete_on_failure=>true, visibility_timeout=>120, max_processing_time=>8000, consumer_threads=>1>
  Error: Unknown error (SystemCallError) - Unknown Error (20109) - C:\Users\user\AppData\Local\Temp\D:Temp20200812-10396-1vyo3pr
  Exception: SystemCallError
  Stack: org/jruby/RubyDir.java:632:in `mkdir'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:89:in `block in mktmpdir'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:137:in `create'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:87:in `mktmpdir'
D:/logstash/logstash-7.8.0-new/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:333:in `block in run_worker_thread'
[2020-08-12T20:52:32,968][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<SystemCallError: Unknown error (SystemCallError) - Unknown Error (20109) - C:\Users\user\AppData\Local\Temp\D:Temp20200812-10396-1vyo3pr>, :backtrace=>["org/jruby/RubyDir.java:632:in `mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:89:in `block in mktmpdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:137:in `create'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tmpdir.rb:87:in `mktmpdir'", "D:/logstash/logstash-7.8.0-new/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:333:in `block in run_worker_thread'"]}
[2020-08-12T20:52:33,002][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
4

Try

temporary_directory        => "Temp"

The ruby Dir.mktmpdir function prefixes the name you pass it with Dir.tmpdir, which in your case appears to be C:\Users\user\AppData\Local\Temp\

5

Now it's working. Thank you so much Badger.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.