S3-plugin logstash "plugin had an unrecoverable error. Will restart this plugin."

Elastic Stack Logstash
1

I am using s3-input plugin to process the logs to logstash, I am receiving some logs of logfiles from s3 but not all of them. As far as I know, my configuration is correct because I am receiving some logs but something is blocking in plugin and getting message as "A plugin had an unrecoverable error. Will restart this plugin"

Here is the error log,

    logstash | Plugin: <LogStash::Inputs::S3 bucket=>"Bucket-name", prefix=>"Bucket-name/", backup_to_bucket=>"Bucket-name", backup_add_prefix=>"Bucket-name-backup/", delete=>true, additional_settings=>{"force_path_style"=>"true", "follow_redirects"=>"false"}, id=>"id-number", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_97f51ba7-9405-4698-9ebc-6d9517772d35", enable_metric=>true, charset=>"UTF-8">, region=>"us-east-1", role_session_name=>"logstash", interval=>60, temporary_directory=>"/tmp/logstash">
logstash    |   Error: 
logstash    |   Exception: Aws::S3::Errors::NotFound
logstash    |   Stack: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/seahorse/client/plugins/response_target.rb:21:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/seahorse/client/request.rb:70:in `send_request'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.67/lib/seahorse/client/base.rb:207:in `block in define_operation_methods'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.67/lib/aws-sdk-resources/request.rb:24:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.67/lib/aws-sdk-resources/operations.rb:41:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.67/lib/aws-sdk-resources/operations.rb:61:in `call'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.67/lib/aws-sdk-resources/resource.rb:147:in `load'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.67/lib/aws-sdk-resources/resource.rb:120:in `data'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.67/lib/aws-sdk-resources/resource.rb:223:in `block in add_data_attribute'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.3.5/lib/logstash/inputs/s3.rb:369:in `process_log'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.3.5/lib/logstash/inputs/s3.rb:161:in `block in process_files'
logstash    | org/jruby/RubyArray.java:1734:in `each'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.3.5/lib/logstash/inputs/s3.rb:156:in `process_files'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.3.5/lib/logstash/inputs/s3.rb:105:in `block in run'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'
logstash    | /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.3.5/lib/logstash/inputs/s3.rb:104:in `run'
logstash    | /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:512:in `inputworker'
logstash    | /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:505:in `block in start_input'
logstash    | [2018-11-13T16:29:42,881][WARN ][logstash.inputs.s3       ] S3 input: Unable to download remote file {:remote_key=>"incapsula/641_827231.log", :message=>"The specified key does not exist."}
logstash    | [2018-11-13T16:39:42,867][WARN ][logstash.inputs.s3       ] S3 input: Unable to download remote file {:remote_key=>"incapsula/641_827259.log", :message=>"The specified key does not exist."}
logstash    | [2018-11-13T16:43:44,305][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.

My Logstash configuration here,

input {
  s3 {
    "bucket" => "bucket-name"
    "prefix" => "bucket-name/"
    "backup_to_bucket" => "bucket-name"
    "backup_add_prefix" => "bucket-name-backup/"
    "delete" => true
    "additional_settings" => {
      "force_path_style" => true
      "follow_redirects" => false
    }
  }
}




output {
       elasticsearch {
        hosts => [ "https://host1:9200", "https://host2:9200", "https://host3:9200" ]
        user => logstash_user
        password => password
        ssl => true
        cacert => '/etc/ca.crt'
        manage_template => false
        index => "incapsula-%{+YYYY.MM.dd}"
    }
    

}
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.