Hi ,
I was trying to collect all my AWS config logs from S3 via logstash and could see it list the filenames fine but once listing is complete it starts to download the S3 file and that fails with an error "Error: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.". Once it gets this error it will restart the plugin and start listing the files again and then error.
Here is the complete log (replaced file paths)
S3 input: Download remote file {:remote_key=>"logs/111/file_brqvFXbF63.json.gz", :local_filename=>"/tmp/logstash/file_brqvFXbF63.json.gz", :level=>:debug, :file=>"logstash/inputs/s3.rb", :line=>"344", :method=>"download_remote_file"}
A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::S3 bucket=>"s3bucket", codec=><LogStash::Codecs::JSON charset=>"UTF-8">, interval=>600, prefix=>"logs/111/", region=>"eu-west-1", use_ssl=>true, delete=>false, temporary_directory=>"/tmp/logstash">
Error: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.
Exception: AWS::S3::Errors::InvalidArgument
Stack: /opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in return_or_raise' /opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:inclient_request'
(eval):3:in get_object' /opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:1371:inget_object'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:1090:in read' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:346:indownload_remote_file'
org/jruby/RubyIO.java:1183:in open' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:345:indownload_remote_file'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:321:in process_log' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:151:inprocess_files'
org/jruby/RubyArray.java:1613:in each' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:146:inprocess_files'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:102:in run' org/jruby/RubyProc.java:281:incall'
/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/interval.rb:20:in interval' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.6/lib/logstash/inputs/s3.rb:101:inrun'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/pipeline.rb:331:in inputworker' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/pipeline.rb:325:instart_input' {:level=>:error, :file=>"logstash/pipeline.rb", :line=>"342", :method=>"inputworker"}
S3 input: Found key {:key=>"logs/111/fil2_akbtQDyyS6nHW.json.gz", :level=>:debug, :file=>"logstash/inputs/s3.rb", :line=>"111", :method=>"list_new_files"}
I can see in logstash-output-s3 plugin "Support S3 v4 signatures" was added some time this year. Does the input plugin has support for AWS S3 v4 signatures?
Regards,
Abey
