# IAM credentials not recognised/used for s3 - plugins used input-s3 + codec-cloudtrail

**URL:** <https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464>\
**Category:** Logstash\
**Created:** [October 19, 2015, 9:35am UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464 "2015-10-19T09:35:16Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\
**Post date:** [October 19, 2015, 9:35am UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/1 "2015-10-19T09:35:16Z")

</div>

Hi,

Scenario is I am trying to use Logstash to:  
-pull AWS CloudTrail '.json.gz' logs from an S3 bucket using [logstash-input-s3](https://rubygems.org/gems/logstash-input-s3) plugin  
-process them using the [logstash-codec-cloudtrail](https://rubygems.org/gems/logstash-codec-cloudtrail) plugin  
-send them to Elasticsearch using standard output.

The installation itself is a standard RPM installation of [logstash-1.5.4-1.noarch.rpm](https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm) with 'logstash-codec-cloudtrail' installed afterward. **Otherwise vanilla.**

The credentials (ACCESS\_KEY + SECRET\_KEY) used exist in the same account as the s3 bucket, and have full access.

I need help with getting Logstash to correctly use/recognise the IAM credentials provided, **because I can confirm that for some reason it does not, and is instead using the IAM Role of the instance.**

I confirmed this by spending hours with AWS Support, who checked the back-end s3 authentication logs, confirmed the role crentials are being used. I spent hours checking the configuration with them and checking the documentation.

This may be a bug, but I'm unsure. I'm trying to make sure I haven't missed something.

This is the exact configuration I'm using:

```
input {
        s3 {
                bucket => "mybucketname-logs-cloudtrail"
                access_key_id => "ACCESS_KEY_HERE"
                secret_access_key => "SECRET_KEY_HERE"
                region => "eu-west-1"
                codec => "cloudtrail"
                type => "cloudtrail"
                prefix => "AWSLogs/AWS_ACCOUNT_ID_HERE/CloudTrail/eu-west-1/2015/09/27"
                temporary_directory => "/tmp/temp-cloudtrail_s3_temp"
                sincedb_path => "/tmp/temp-cloudtrail_s3_sincedb"
                debug => "true"
        }
}
output {
        elasticsearch {
                host => "ELASTICSEARCH_URL_HERE"
                protocol => "http"
        }
        stdout {
                codec => "rubydebug"
        }
}

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 20, 2015, 5:45am UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/2 "2015-10-20T05:45:57Z")

</div>

Have you run this with `-v` or `--debug` to get more information with what is happening in the LS side of things?

---

<div class="post-metadata">

**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\
**Post date:** [October 20, 2015, 9:06am UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/3 "2015-10-20T09:06:14Z")

</div>

With the assistance of the AWS staff, I was able to enable Ruby debug.

From the investigations we did, it is apparent that Logstash is just simply not using the provided credentials, and is instead prioritising the IAM role.

This is incorrect behaviour as far as I'm aware, especially in the [AWS Ruby Developer Guide](http://docs.aws.amazon.com/AWSSdkDocsRuby/latest//DeveloperGuide/prog-basics-creds.html):

> It looks for default credentials in the following order and loads the first set that it finds:

> 1. Credentials passed to the AWS.config method with the :access\_key\_id and :secret\_access\_key\_id options.
> 2. Environment Variables – AWS\_ACCESS\_KEY\_ID and AWS\_SECRET\_ACCESS\_KEY environment variables.  
> The SDK for Ruby uses the ENVProvider class to load these credentials.
> 3. The credentials file's default profile – For more information about the credentials file, see Setting up AWS Credentials.  
> The SDK for Ruby uses the SharedCredentialFileProvider to load profiles.
> 4. Instance profile credentials – these credentials can be assigned to Amazon EC2 instances, and are delivered through the Amazon EC2 metadata service.  
> The SDK for Ruby uses EC2Provider to load these credentials.

I need to find out how to fix or override this function, I think.

FYI: I'm not a Ruby Developer.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 20, 2015, 9:25am UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/4 "2015-10-20T09:25:52Z")

</div>

If you can show us some debug information it'll help.

---

<div class="post-metadata">

**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\
**Post date:** [October 20, 2015, 1:14pm UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/5 "2015-10-20T13:14:18Z")

</div>

Is there anything in particular that is useful?

I have reams and reams of logs, but I dont think that will be useful as most of it is 'Access Denied' errors.

Thanks.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 20, 2015, 9:00pm UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/6 "2015-10-20T21:00:54Z")

</div>

If you can gist/pastebin/etc some of it and link here I can ask a dev to take a look and we can go from there.

---

<div class="post-metadata">

**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\
**Post date:** [October 21, 2015, 12:04pm UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/7 "2015-10-21T12:04:10Z")

</div>

Hi Mark / @warkolm ,

Thanks for getting back to me.  
I'm going to try and get some details together about this and will add them here.

Kareem.

---

<div class="post-metadata">

**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\
**Post date:** [October 22, 2015, 7:41pm UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/8 "2015-10-22T19:41:47Z")

</div>

Hi Mark @warkolm ,

Upon further testing, its definitely not using the credentials from the config file.

We tested this by giving both the IAM User (In config) and the IAM Role (For the instance) the same permissions to S3, then removing the S3 permissions for the Role.

Also, to take the variable of cross-account access out, this was tested to an S3 bucket local to the instances' account.

We added the following code to the **input/s3.rb** file:

```
s3 = AWS::S3.new(:logger => Logger.new($stdout), :http_wire_trace => true)

```

Then tailed out the **logstash.stdout** log file, in both Permission scenarios.

The rest will follow in the next response below.

---

<div class="post-metadata">

**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\
**Post date:** [October 22, 2015, 7:42pm UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/9 "2015-10-22T19:42:13Z")

</div>

@warkolm

Please note the following 2 responses:

**SUCCESSFUL RESPONSE:**

> \<- "HEAD /AWSLogs/327295020676/CloudTrail/eu-west-1/2015/09/26/XXXAWSACCOUNTNUMBER\_CloudTrail\_eu-west-1\_20150926T2135Z\_VHDFW9zDUgsb4ZSg.json.gz HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby/1.64.0 jruby/1.9.3 java\r\nX-Amz-Security-Token: AQoDYXdzENv//////////wEa4APUd0y/Mqp6GQ5ZA1CFeE7IzNWKC6xxvcCiLA2Gr64nxFiKDNyfNF9CT2XLYc1t3KNSktZOmKG45fm5soNZ/JbsPjtvyyau5xG/EWGUUJSvMnA++nEJxiFz6Ki5vsSIdRe9FmNMRtETFhiJfZjcPd50AM09ryVLwDzq1b+c7tJACrE6YjO5J8k/Obi+D5wRO9SsvkDkvjxr6Li2qpV2rXtLnPEsTxrvuSUkZILnBK2FoAwtUg6eMwhdn8sGQCEAjHtGpayC2rBJguJCiD93lvREPyTngj/U4FF+a9fdoZnfxnIoPnK07JcK4iZz/7IlwL9KX0+C5sU2q4DFyOiVTIIO9xwriKNmZ29jbnEbpLUoYSEDTiRQV3GXtg7p7p92ptZx5j91LcoZ9rpHpD0dTpUmtF7KxeC3nvCKpq1H/ZHXwxjyAacLJc9z9j7gyB4uZx5qB95Q7xlJzw9VM3kx5MqCVS9FOsGiJCa78pg4XY2iAk476Y6K8Sdyme7Of2kckcqj1UYAQhoDz56nbKPKfbG0/2mdxcn491a1SBCE4qbcIvQj465q7Hfn4LqtyvIgEwI3DOZCEznquWkk4822UYbsu4r5FPYaburQ0tRgDnsiP0pin++m8YTa7Q7MY630x2gg082ksQU=\r\nDate: Thu, 22 Oct 2015 19:14:37 GMT\r\nAuthorization: AWS ASIAI4VF72XFAJV3A2JQ:Jt/0N8qLD45dQUoNSKMIgHe1meE=\r\nAccept: _/_\r\nHost: [my-bucket-name.s3.amazonaws.com](http://my-bucket-name.s3.amazonaws.com)\r\n\r\n"  
> -\> "HTTP/1.1 200 OK\r\n"  
> -\> "x-amz-id-2: MwhGNm2BgVcfYFc1dmzWNTeNeU1Eq0xyld04FtW+1SS9sPHaPszF2sRNhhHg5rXm7blYecYYVPE=\r\n"  
> -\> "x-amz-request-id: 95F53DE39BA25FC7\r\n"  
> -\> "Date: Thu, 22 Oct 2015 19:14:38 GMT\r\n"  
> -\> "Content-Encoding: gzip\r\n"  
> -\> "Last-Modified: Sat, 26 Sep 2015 21:33:53 GMT\r\n"  
> -\> "ETag: "384145b2dfd3919b544995da4ca29562"\r\n"  
> -\> "x-amz-server-side-encryption: AES256\r\n"  
> -\> "Accept-Ranges: bytes\r\n"  
> -\> "Content-Type: application/json\r\n"  
> -\> "Content-Length: 6849\r\n"  
> -\> "Server: AmazonS3\r\n"  
> -\> "\r\n"  
> Conn keep-alive  
> I, [2015-10-22T19:14:37.312000 #3477] INFO -- : [AWS S3 200 0.033 0 retries] head\_object(:bucket\_name=\>"my-bucket-name",:key=\>"AWSLogs/XXXAWSACCOUNTNUMBER/CloudTrail/eu-west-1/2015/09/26/XXXAWSACCOUNTNUMBER\_CloudTrail\_eu-west-1\_20150926T2135Z\_VHDFW9zDUgsb4ZSg.json.gz")

**ERROR RESPONSE:**

> ^X\<- "GET /?max-keys=1000&prefix=AWSLogs%2FXXXAWSACCOUNTNUMBER%2FCloudTrail%2Feu-west-1%2F2015%2F09%2F26 HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby/1.64.0 jruby/1.9.3 java\r\nX-Amz-Security-Token: AQoDYXdzENv//////////wEa4APUd0y/Mqp6GQ5ZA1CFeE7IzNWKC6xxvcCiLA2Gr64nxFiKDNyfNF9CT2XLYc1t3KNSktZOmKG45fm5soNZ/JbsPjtvyyau5xG/EWGUUJSvMnA++nEJxiFz6Ki5vsSIdRe9FmNMRtETFhiJfZjcPd50AM09ryVLwDzq1b+c7tJACrE6YjO5J8k/Obi+D5wRO9SsvkDkvjxr6Li2qpV2rXtLnPEsTxrvuSUkZILnBK2FoAwtUg6eMwhdn8sGQCEAjHtGpayC2rBJguJCiD93lvREPyTngj/U4FF+a9fdoZnfxnIoPnK07JcK4iZz/7IlwL9KX0+C5sU2q4DFyOiVTIIO9xwriKNmZ29jbnEbpLUoYSEDTiRQV3GXtg7p7p92ptZx5j91LcoZ9rpHpD0dTpUmtF7KxeC3nvCKpq1H/ZHXwxjyAacLJc9z9j7gyB4uZx5qB95Q7xlJzw9VM3kx5MqCVS9FOsGiJCa78pg4XY2iAk476Y6K8Sdyme7Of2kckcqj1UYAQhoDz56nbKPKfbG0/2mdxcn491a1SBCE4qbcIvQj465q7Hfn4LqtyvIgEwI3DOZCEznquWkk4822UYbsu4r5FPYaburQ0tRgDnsiP0pin++m8YTa7Q7MY630x2gg082ksQU=\r\nDate: Thu, 22 Oct 2015 19:16:07 GMT\r\nAuthorization: AWS ASIAI4VF72XFAJV3A2JQ:65/EvOou4N7k53P95P5r38SPylY=\r\nAccept: _/_\r\nHost: [my-bucket-name.s3.amazonaws.com](http://my-bucket-name.s3.amazonaws.com)\r\n\r\n"  
> -\> "HTTP/1.1 403 Forbidden\r\n"  
> -\> "x-amz-bucket-region: eu-west-1\r\n"  
> -\> "x-amz-request-id: 5AAA1405488AFF7D\r\n"  
> -\> "x-amz-id-2: vzyuecquJEdWGofAgC9XjcEtVDDbz7+f19ghsEou3dScAUJ5SS6FjvqhiyEBiMwJiXdBAhbP7K8=\r\n"  
> -\> "Content-Type: application/xml\r\n"  
> -\> "Transfer-Encoding: chunked\r\n"  
> -\> "Date: Thu, 22 Oct 2015 19:16:07 GMT\r\n"  
> -\> "Server: AmazonS3\r\n"  
> -\> "\r\n"  
> -\> "f3\r\n"  
> reading 243 bytes...  
> -\> "\<?xml version=\"1.0\" encoding=\"UTF-8\"?\>\n`AccessDenied`Access Denied5AAA1405488AFF7DvzyuecquJEdWGofAgC9XjcEtVDDbz7+f19ghsEou3dScAUJ5SS6FjvqhiyEBiMwJiXdBAhbP7K8="  
> read 243 bytes  
> reading 2 bytes...  
> -\> "\r\n"  
> read 2 bytes  
> -\> "0\r\n"  
> -\> "\r\n"  
> Conn keep-alive  
> I, [2015-10-22T19:16:07.760000 #3477] INFO -- : [AWS S3 403 0.042 0 retries] list\_objects(:bucket\_name=\>"evision-cloudtrail",:max\_keys=\>1000,:prefix=\>"AWSLogs/XXXAWSACCOUNTNUMBER/CloudTrail/eu-west-1/2015/09/26") AWS::S3::Errors::AccessDenied Access Denied

I suspect that there may be something up with this file:  
**/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-mixin-aws-1.0.1/lib/logstash/plugin\_mixins/aws\_config/v1.rb**

Especially, as [this enhancement](https://github.com/elastic/logstash/issues/1778) has some discussion surrounding defaulting to IAM roles.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 22, 2015, 11:11pm UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/10 "2015-10-22T23:11:11Z")

</div>

Do you have the logs from `--debug` as well? If you do please put into gist/pastebin/etc and link 🙂

---

<div class="post-metadata">

**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\
**Post date:** [October 26, 2015, 9:42am UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/11 "2015-10-26T09:42:03Z")

</div>

Hi Mark @warkolm,

I have enabled '--debug' in /etc/init.d/logstash and pasted some of the output here:

[http://pastebin.com/cw6BpnhR](http://pastebin.com/cw6BpnhR)

Kareem.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 27, 2015, 1:06am UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/12 "2015-10-27T01:06:11Z")

</div>

It looks like this is an actual bug and I've raised this as a summary [here](https://github.com/logstash-plugins/logstash-input-s3/issues/60).

Thanks for this extra info and helping us get to the bottom of it!

---

<div class="post-metadata">

**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\
**Post date:** [October 27, 2015, 9:27am UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/13 "2015-10-27T09:27:12Z")

</div>

Hi Mark @warkolm

Thanks for submitting this as a bug, but I am hoping that there may be some  
workaround that we can determine to get my Logstash instance working as  
intended.

I have been at this for a few weeks now, so it's becoming quite a blocker  
for some elements of my current project.

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:25am UTC](https://discuss.elastic.co/t/iam-credentials-not-recognised-used-for-s3-plugins-used-input-s3-codec-cloudtrail/32464/14 "2017-07-06T05:25:17Z")

</div>