Hi!
I have an ec2 instance in eu-west-1a, an IAM role attached and I can see the temporary creds if I do:
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
I have a small Python program to test if the temporary creds work, which looks like this:
import boto3
import botocore
BUCKET_NAME = 'bucket-name' # replace with your bucket name
KEY = 'AWSLogs/TestFile' # replace with your object key
s3 = boto3.resource('s3')
try:
s3.Bucket(BUCKET_NAME).download_file(KEY, 'TestFile')
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == "404":
print("The object does not exist.")
else:
raise
Running the above script in the ec2 instance works fine, it's definitely using the temporary creds as there are no environment variable set nor any .aws/credentials set either.
I have also tested that the IAM role can access the bucket using the policy simulator, which it can.
However using the s3 input plugin I get Access Denied errors. Below is my config and then the errors:
input {
s3 {
bucket => "bucket-name"
region => "eu-central-1"
role_arn => "full arn of the IAM role attached to this instance"
role_session_name => "mysession"
sincedb_path => "/var/lib/logstash/sincedb"
type => "s3"
}
}
[2018-10-09T11:04:36,568][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::S3 bucket=>\"bucket-name\", role_arn=>\"arn:aws:iam::999999999:role/rolename\", role_session_name=>\"mysession\", id=>\"eaaa92276690fd9e6d3fd8bb421e70670ebc78afe4010fd110f0c62bd6b54de2\", region=>\"eu-central-1\", sincedb_path=>\"/var/lib/logstash/sincedb\", type=>\"s3\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_ce7e1261-1b24-44c9-9b95-fe4d8df9d50f\", enable_metric=>true, charset=>\"UTF-8\">, delete=>false, interval=>60, temporary_directory=>\"/tmp/logstash\", include_object_properties=>false>", :error=>"Access denied", :thread=>"#<Thread:0x691ea2a9 run>"}
[2018-10-09T11:04:38,062][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Aws::STS::Errors::AccessDenied: Access denied>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:incall'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/aws-sdk-core/plugins/param_converter.rb:20:incall'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/seahorse/client/plugins/response_target.rb:21:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/seahorse/client/request.rb:70:insend_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/seahorse/client/base.rb:207:in block in define_operation_methods'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/aws-sdk-core/assume_role_credentials.rb:49:inrefresh'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/aws-sdk-core/refreshing_credentials.rb:20:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.104/lib/aws-sdk-core/assume_role_credentials.rb:40:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-aws-4.3.0/lib/logstash/plugin_mixins/aws_config/v2.rb:69:in assume_role'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-aws-4.3.0/lib/logstash/plugin_mixins/aws_config/v2.rb:63:incredentials'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-aws-4.3.0/lib/logstash/plugin_mixins/aws_config/v2.rb:18:in aws_options_hash'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.3.7/lib/logstash/inputs/s3.rb:423:inget_s3object'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.3.7/lib/logstash/inputs/s3.rb:86:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:241:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:252:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:252:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:395:instart_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:293:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:199:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:159:in `block in start'"], :thread=>"#<Thread:0x691ea2a9 run>"}
[2018-10-09T11:04:38,077][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
Could anyone suggest a way of determining what the input plugin is actually doing, what creds it's trying to use, to try and get to the bottom of the problem.
Many thanks