# Logstash S3 Input Plugin not recognizing IAM Instance Profile

**URL:** <https://discuss.elastic.co/t/logstash-s3-input-plugin-not-recognizing-iam-instance-profile/259812>\
**Category:** Logstash\
**Created:** [December 29, 2020, 4:28pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-not-recognizing-iam-instance-profile/259812 "2020-12-29T16:28:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![prod](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prod/32/81509_2.png) [@prod](https://discuss.elastic.co/u/prod)\
**Post date:** [December 29, 2020, 4:28pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-not-recognizing-iam-instance-profile/259812/1 "2020-12-29T16:28:21Z")

</div>

Checking the S3 Input plugin documentation

```auto
This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order:

    Static configuration, using access_key_id and secret_access_key params in logstash plugin config
    External credentials file specified by aws_credentials_file
    Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
    Environment variables AMAZON_ACCESS_KEY_ID and AMAZON_SECRET_ACCESS_KEY
    IAM Instance Profile (available when running inside EC2)

```

The EC2 instance where Logstash is running currently has an Instance Profile set up that allows the access to read files from an S3 bucket. I also have a role that allows the instance to assume it and read the S3 bucket contents.

The problem is that if I configure the S3 input plugin to rely on it instead of user credentials, I get the following error:

**S3 INPUT PLUGIN SETUP**

```auto
input {
  s3 {
    id => "s3-input-cloudtrail"
    bucket => "mycompany-cloudtrail-logs"
    prefix => "AWSLogs/companyawsid/CloudTrail/"
    codec => "json"
    # With or without the next line commented, I get the same error message
    #role_arn => "arn:aws:iam::companyawsid:role/LogstashS3ReadAccess"
  }
}

```

**ERROR**

```auto
[2020-12-29T16:21:24,640][ERROR][logstash.javapipeline][aws-cloudtrail][s3-input-cloudtrail] A plugin had an unrecoverable error. Wil
l restart this plugin.
  Pipeline_id:aws-cloudtrail
  Plugin: <LogStash::Inputs::S3 bucket=>"mycompany-cloudtrail-logs", codec=><LogStash::Codecs::JSON id=>"json_7c4671e2-8cea-49f7-a8a9-08
b0bbe0f099", enable_metric=>true, charset=>"UTF-8">, id=>"s3-input-cloudtrail", prefix=>"AWSLogs/companyawsid/CloudTrail/", enable_metric
=>true, region=>"us-east-1", role_session_name=>"logstash", delete=>false, interval=>60, watch_for_new_files=>true, temporary_directory=>
"/tmp/logstash", include_object_properties=>false, gzip_pattern=>".gz(ip)?$">
  Error: unable to sign request without credentials set
  Exception: Aws::Errors::MissingCredentialsError
  Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/request_signer.rb:104:in `requ
ire_credentials'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_request_signer.rb:14:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_host_id.rb:14:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/retry_errors.rb:108:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_md5s.rb:31:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_iad_regional_endpoint.rb:31:in `call
'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:in `c
all'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/rest/handler.rb:7:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/endpoint_pattern.rb:27:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/endpoint_discovery.rb:67:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/plugins/endpoint.rb:41:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call
'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/request.rb:70:in `send_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.605/lib/seahorse/client/base.rb:207:in `block in define_operation_me
thods'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.605/lib/aws-sdk-resources/request.rb:24:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.605/lib/aws-sdk-resources/operations.rb:139:in `all_batches'
org/jruby/RubyEnumerator.java:396:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.605/lib/aws-sdk-resources/collection.rb:18:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:132:in `list_new_files'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:172:in `process_files'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:123:in `block in run'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:122:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'

```

What might be the cause of this issue?

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [December 29, 2020, 8:53pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-not-recognizing-iam-instance-profile/259812/2 "2020-12-29T20:53:50Z")

</div>

Related Thread ans Issue

> [@IAM credentials not recognised/used for s3 - plugins used input-s3 + codec-cloudtrail](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464):
>
> Hi, Scenario is I am trying to use Logstash to: -pull AWS CloudTrail '.json.gz' logs from an S3 bucket using [logstash-input-s3](https://rubygems.org/gems/logstash-input-s3) plugin -process them using the [logstash-codec-cloudtrail](https://rubygems.org/gems/logstash-codec-cloudtrail) plugin -send them to Elasticsearch using standard output. The installation itself is a standard RPM installation of [logstash-1.5.4-1.noarch.rpm](https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm) with 'logstash-codec-cloudtrail' installed afterward. Otherwise vanilla. The credentials (ACCESS\_KEY + SECRET\_KEY) used exist in the same account as the s3 bucket, an…

> <https://github.com/logstash-plugins/logstash-input-s3/issues/60>
>
> This was initially reported in a Discuss thread \[here\](https://discuss.elastic.c…o/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464).
> 
> Config;
> 
> \`\`\`
> input {
> s3 {
> bucket =\> "mybucketname-logs-cloudtrail"
> access\_key\_id =\> "ACCESS\_KEY\_HERE"
> secret\_access\_key =\> "SECRET\_KEY\_HERE"
> region =\> "eu-west-1"
> codec =\> "cloudtrail"
> type =\> "cloudtrail"
> prefix =\> "AWSLogs/AWS\_ACCOUNT\_ID\_HERE/CloudTrail/eu-west-1/2015/09/27"
> temporary\_directory =\> "/tmp/temp-cloudtrail\_s3\_temp"
> sincedb\_path =\> "/tmp/temp-cloudtrail\_s3\_sincedb"
> debug =\> "true"
> }
> }
> output {
> elasticsearch {
> host =\> "ELASTICSEARCH\_URL\_HERE"
> protocol =\> "http"
> }
> stdout {
> codec =\> "rubydebug"
> }
> }
> \`\`\`
> 
> There's some debug logging \[here\](http://pastebin.com/cw6BpnhR) as well.
> 
> I hope the discuss thread and this gives you a good idea of the situation, but let me know if my summary sucks :)

---

<div class="post-metadata">

**Author:** ![prod](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prod/32/81509_2.png) [@prod](https://discuss.elastic.co/u/prod)\
**Post date:** [December 30, 2020, 12:47pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-not-recognizing-iam-instance-profile/259812/3 "2020-12-30T12:47:44Z")

</div>

Hmm, it seems related at an SDK level, but the provided ticket refers to AWS credentials (through access/secret keys) not being recognized.

In my specific case the contrary is happening: The access/secret keys are being properly recognized if I set them up, but the IAM role is not even if I specify the ARN. The idea is being able to use roles instead of user credentials.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 27, 2021, 12:48pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-not-recognizing-iam-instance-profile/259812/4 "2021-01-27T12:48:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.