Logstash S3 Input Plugin not recognizing IAM Instance Profile

Checking the S3 Input plugin documentation

This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order:

    Static configuration, using access_key_id and secret_access_key params in logstash plugin config
    External credentials file specified by aws_credentials_file
    Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
    Environment variables AMAZON_ACCESS_KEY_ID and AMAZON_SECRET_ACCESS_KEY
    IAM Instance Profile (available when running inside EC2)

The EC2 instance where Logstash is running currently has an Instance Profile set up that allows the access to read files from an S3 bucket. I also have a role that allows the instance to assume it and read the S3 bucket contents.

The problem is that if I configure the S3 input plugin to rely on it instead of user credentials, I get the following error:

S3 INPUT PLUGIN SETUP

input {
  s3 {
    id => "s3-input-cloudtrail"
    bucket => "mycompany-cloudtrail-logs"
    prefix => "AWSLogs/companyawsid/CloudTrail/"
    codec => "json"
    # With or without the next line commented, I get the same error message
    #role_arn => "arn:aws:iam::companyawsid:role/LogstashS3ReadAccess"
  }
}

ERROR

[2020-12-29T16:21:24,640][ERROR][logstash.javapipeline    ][aws-cloudtrail][s3-input-cloudtrail] A plugin had an unrecoverable error. Wil
l restart this plugin.
  Pipeline_id:aws-cloudtrail
  Plugin: <LogStash::Inputs::S3 bucket=>"mycompany-cloudtrail-logs", codec=><LogStash::Codecs::JSON id=>"json_7c4671e2-8cea-49f7-a8a9-08
b0bbe0f099", enable_metric=>true, charset=>"UTF-8">, id=>"s3-input-cloudtrail", prefix=>"AWSLogs/companyawsid/CloudTrail/", enable_metric
=>true, region=>"us-east-1", role_session_name=>"logstash", delete=>false, interval=>60, watch_for_new_files=>true, temporary_directory=>
"/tmp/logstash", include_object_properties=>false, gzip_pattern=>".gz(ip)?$">
  Error: unable to sign request without credentials set
  Exception: Aws::Errors::MissingCredentialsError
  Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/request_signer.rb:104:in `requ
ire_credentials'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_request_signer.rb:14:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_host_id.rb:14:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/retry_errors.rb:108:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_md5s.rb:31:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_iad_regional_endpoint.rb:31:in `call
'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:in `c
all'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/rest/handler.rb:7:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/endpoint_pattern.rb:27:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/endpoint_discovery.rb:67:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/plugins/endpoint.rb:41:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call
'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/request.rb:70:in `send_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/base.rb:207:in `block in define_operation_me
thods'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.605/lib/aws-sdk-resources/request.rb:24:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.605/lib/aws-sdk-resources/operations.rb:139:in `all_batches'
org/jruby/RubyEnumerator.java:396:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.605/lib/aws-sdk-resources/collection.rb:18:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:132:in `list_new_files'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:172:in `process_files'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:123:in `block in run'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:122:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'

What might be the cause of this issue?

Related Thread ans Issue

Hmm, it seems related at an SDK level, but the provided ticket refers to AWS credentials (through access/secret keys) not being recognized.

In my specific case the contrary is happening: The access/secret keys are being properly recognized if I set them up, but the IAM role is not even if I specify the ARN. The idea is being able to use roles instead of user credentials.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.