IAM credentials not recognised/used for s3 - plugins used input-s3 + codec-cloudtrail

[plonka2000 (Kareem)]

Hi,

Scenario is I am trying to use Logstash to:
-pull AWS CloudTrail '.json.gz' logs from an S3 bucket using logstash-input-s3 plugin
-process them using the logstash-codec-cloudtrail plugin
-send them to Elasticsearch using standard output.

The installation itself is a standard RPM installation of logstash-1.5.4-1.noarch.rpm with 'logstash-codec-cloudtrail' installed afterward. Otherwise vanilla.

The credentials (ACCESS_KEY + SECRET_KEY) used exist in the same account as the s3 bucket, and have full access.

I need help with getting Logstash to correctly use/recognise the IAM credentials provided, because I can confirm that for some reason it does not, and is instead using the IAM Role of the instance.

I confirmed this by spending hours with AWS Support, who checked the back-end s3 authentication logs, confirmed the role crentials are being used. I spent hours checking the configuration with them and checking the documentation.

This may be a bug, but I'm unsure. I'm trying to make sure I haven't missed something.

This is the exact configuration I'm using:

input {
        s3 {
                bucket              => "mybucketname-logs-cloudtrail"
                access_key_id       => "ACCESS_KEY_HERE"
                secret_access_key   => "SECRET_KEY_HERE"
                region              => "eu-west-1"
                codec               => "cloudtrail"
                type                => "cloudtrail"
                prefix              => "AWSLogs/AWS_ACCOUNT_ID_HERE/CloudTrail/eu-west-1/2015/09/27"
                temporary_directory => "/tmp/temp-cloudtrail_s3_temp"
                sincedb_path        => "/tmp/temp-cloudtrail_s3_sincedb"
                debug               => "true"
        }
}
output {
        elasticsearch {
                host => "ELASTICSEARCH_URL_HERE"
                protocol => "http"
        }
        stdout {
                codec => "rubydebug"
        }
}

[warkolm (Mark Walkom)]

Have you run this with -v or --debug to get more information with what is happening in the LS side of things?

[plonka2000 (Kareem)]

With the assistance of the AWS staff, I was able to enable Ruby debug.

From the investigations we did, it is apparent that Logstash is just simply not using the provided credentials, and is instead prioritising the IAM role.

This is incorrect behaviour as far as I'm aware, especially in the AWS Ruby Developer Guide:

It looks for default credentials in the following order and loads the first set that it finds:

1. Credentials passed to the AWS.config method with the :access_key_id and :secret_access_key_id options.
2. Environment Variables – AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
   The SDK for Ruby uses the ENVProvider class to load these credentials.
3. The credentials file's default profile – For more information about the credentials file, see Setting up AWS Credentials.
   The SDK for Ruby uses the SharedCredentialFileProvider to load profiles.
4. Instance profile credentials – these credentials can be assigned to Amazon EC2 instances, and are delivered through the Amazon EC2 metadata service.
   The SDK for Ruby uses EC2Provider to load these credentials.

I need to find out how to fix or override this function, I think.

FYI: I'm not a Ruby Developer.

[warkolm (Mark Walkom)]

If you can show us some debug information it'll help.

[plonka2000 (Kareem)]

Is there anything in particular that is useful?

I have reams and reams of logs, but I dont think that will be useful as most of it is 'Access Denied' errors.

Thanks.

[warkolm (Mark Walkom)]

If you can gist/pastebin/etc some of it and link here I can ask a dev to take a look and we can go from there.

[plonka2000 (Kareem)]

Hi Mark / @warkolm ,

Thanks for getting back to me.
I'm going to try and get some details together about this and will add them here.

Kareem.

[plonka2000 (Kareem)]

Hi Mark @warkolm ,

Upon further testing, its definitely not using the credentials from the config file.

We tested this by giving both the IAM User (In config) and the IAM Role (For the instance) the same permissions to S3, then removing the S3 permissions for the Role.

Also, to take the variable of cross-account access out, this was tested to an S3 bucket local to the instances' account.

We added the following code to the input/s3.rb file:

s3 = AWS::S3.new(:logger => Logger.new($stdout), :http_wire_trace => true)

Then tailed out the logstash.stdout log file, in both Permission scenarios.

The rest will follow in the next response below.

[plonka2000 (Kareem)]

@warkolm

Please note the following 2 responses:

SUCCESSFUL RESPONSE:

<- "HEAD /AWSLogs/327295020676/CloudTrail/eu-west-1/2015/09/26/XXXAWSACCOUNTNUMBER_CloudTrail_eu-west-1_20150926T2135Z_VHDFW9zDUgsb4ZSg.json.gz HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby/1.64.0 jruby/1.9.3 java\r\nX-Amz-Security-Token: AQoDYXdzENv//////////wEa4APUd0y/Mqp6GQ5ZA1CFeE7IzNWKC6xxvcCiLA2Gr64nxFiKDNyfNF9CT2XLYc1t3KNSktZOmKG45fm5soNZ/JbsPjtvyyau5xG/EWGUUJSvMnA++nEJxiFz6Ki5vsSIdRe9FmNMRtETFhiJfZjcPd50AM09ryVLwDzq1b+c7tJACrE6YjO5J8k/Obi+D5wRO9SsvkDkvjxr6Li2qpV2rXtLnPEsTxrvuSUkZILnBK2FoAwtUg6eMwhdn8sGQCEAjHtGpayC2rBJguJCiD93lvREPyTngj/U4FF+a9fdoZnfxnIoPnK07JcK4iZz/7IlwL9KX0+C5sU2q4DFyOiVTIIO9xwriKNmZ29jbnEbpLUoYSEDTiRQV3GXtg7p7p92ptZx5j91LcoZ9rpHpD0dTpUmtF7KxeC3nvCKpq1H/ZHXwxjyAacLJc9z9j7gyB4uZx5qB95Q7xlJzw9VM3kx5MqCVS9FOsGiJCa78pg4XY2iAk476Y6K8Sdyme7Of2kckcqj1UYAQhoDz56nbKPKfbG0/2mdxcn491a1SBCE4qbcIvQj465q7Hfn4LqtyvIgEwI3DOZCEznquWkk4822UYbsu4r5FPYaburQ0tRgDnsiP0pin++m8YTa7Q7MY630x2gg082ksQU=\r\nDate: Thu, 22 Oct 2015 19:14:37 GMT\r\nAuthorization: AWS ASIAI4VF72XFAJV3A2JQ:Jt/0N8qLD45dQUoNSKMIgHe1meE=\r\nAccept: /\r\nHost: my-bucket-name.s3.amazonaws.com\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "x-amz-id-2: MwhGNm2BgVcfYFc1dmzWNTeNeU1Eq0xyld04FtW+1SS9sPHaPszF2sRNhhHg5rXm7blYecYYVPE=\r\n"
-> "x-amz-request-id: 95F53DE39BA25FC7\r\n"
-> "Date: Thu, 22 Oct 2015 19:14:38 GMT\r\n"
-> "Content-Encoding: gzip\r\n"
-> "Last-Modified: Sat, 26 Sep 2015 21:33:53 GMT\r\n"
-> "ETag: "384145b2dfd3919b544995da4ca29562"\r\n"
-> "x-amz-server-side-encryption: AES256\r\n"
-> "Accept-Ranges: bytes\r\n"
-> "Content-Type: application/json\r\n"
-> "Content-Length: 6849\r\n"
-> "Server: AmazonS3\r\n"
-> "\r\n"
Conn keep-alive
I, [2015-10-22T19:14:37.312000 #3477] INFO -- : [AWS S3 200 0.033 0 retries] head_object(:bucket_name=>"my-bucket-name",:key=>"AWSLogs/XXXAWSACCOUNTNUMBER/CloudTrail/eu-west-1/2015/09/26/XXXAWSACCOUNTNUMBER_CloudTrail_eu-west-1_20150926T2135Z_VHDFW9zDUgsb4ZSg.json.gz")

ERROR RESPONSE:

^X<- "GET /?max-keys=1000&prefix=AWSLogs%2FXXXAWSACCOUNTNUMBER%2FCloudTrail%2Feu-west-1%2F2015%2F09%2F26 HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby/1.64.0 jruby/1.9.3 java\r\nX-Amz-Security-Token: AQoDYXdzENv//////////wEa4APUd0y/Mqp6GQ5ZA1CFeE7IzNWKC6xxvcCiLA2Gr64nxFiKDNyfNF9CT2XLYc1t3KNSktZOmKG45fm5soNZ/JbsPjtvyyau5xG/EWGUUJSvMnA++nEJxiFz6Ki5vsSIdRe9FmNMRtETFhiJfZjcPd50AM09ryVLwDzq1b+c7tJACrE6YjO5J8k/Obi+D5wRO9SsvkDkvjxr6Li2qpV2rXtLnPEsTxrvuSUkZILnBK2FoAwtUg6eMwhdn8sGQCEAjHtGpayC2rBJguJCiD93lvREPyTngj/U4FF+a9fdoZnfxnIoPnK07JcK4iZz/7IlwL9KX0+C5sU2q4DFyOiVTIIO9xwriKNmZ29jbnEbpLUoYSEDTiRQV3GXtg7p7p92ptZx5j91LcoZ9rpHpD0dTpUmtF7KxeC3nvCKpq1H/ZHXwxjyAacLJc9z9j7gyB4uZx5qB95Q7xlJzw9VM3kx5MqCVS9FOsGiJCa78pg4XY2iAk476Y6K8Sdyme7Of2kckcqj1UYAQhoDz56nbKPKfbG0/2mdxcn491a1SBCE4qbcIvQj465q7Hfn4LqtyvIgEwI3DOZCEznquWkk4822UYbsu4r5FPYaburQ0tRgDnsiP0pin++m8YTa7Q7MY630x2gg082ksQU=\r\nDate: Thu, 22 Oct 2015 19:16:07 GMT\r\nAuthorization: AWS ASIAI4VF72XFAJV3A2JQ:65/EvOou4N7k53P95P5r38SPylY=\r\nAccept: /\r\nHost: my-bucket-name.s3.amazonaws.com\r\n\r\n"
-> "HTTP/1.1 403 Forbidden\r\n"
-> "x-amz-bucket-region: eu-west-1\r\n"
-> "x-amz-request-id: 5AAA1405488AFF7D\r\n"
-> "x-amz-id-2: vzyuecquJEdWGofAgC9XjcEtVDDbz7+f19ghsEou3dScAUJ5SS6FjvqhiyEBiMwJiXdBAhbP7K8=\r\n"
-> "Content-Type: application/xml\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Date: Thu, 22 Oct 2015 19:16:07 GMT\r\n"
-> "Server: AmazonS3\r\n"
-> "\r\n"
-> "f3\r\n"
reading 243 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\nAccessDeniedAccess Denied5AAA1405488AFF7DvzyuecquJEdWGofAgC9XjcEtVDDbz7+f19ghsEou3dScAUJ5SS6FjvqhiyEBiMwJiXdBAhbP7K8="
read 243 bytes
reading 2 bytes...
-> "\r\n"
read 2 bytes
-> "0\r\n"
-> "\r\n"
Conn keep-alive
I, [2015-10-22T19:16:07.760000 #3477] INFO -- : [AWS S3 403 0.042 0 retries] list_objects(:bucket_name=>"evision-cloudtrail",:max_keys=>1000,:prefix=>"AWSLogs/XXXAWSACCOUNTNUMBER/CloudTrail/eu-west-1/2015/09/26") AWS::S3::Errors::AccessDenied Access Denied

I suspect that there may be something up with this file:
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-mixin-aws-1.0.1/lib/logstash/plugin_mixins/aws_config/v1.rb

Especially, as this enhancement has some discussion surrounding defaulting to IAM roles.

[warkolm (Mark Walkom)]

Do you have the logs from --debug as well? If you do please put into gist/pastebin/etc and link

[plonka2000 (Kareem)]

Hi Mark @warkolm,

I have enabled '--debug' in /etc/init.d/logstash and pasted some of the output here:

http://pastebin.com/cw6BpnhR

Kareem.

[warkolm (Mark Walkom)]

It looks like this is an actual bug and I've raised this as a summary here.

Thanks for this extra info and helping us get to the bottom of it!

[plonka2000 (Kareem)]

Hi Mark @warkolm

Thanks for submitting this as a bug, but I am hoping that there may be some
workaround that we can determine to get my Logstash instance working as
intended.

I have been at this for a few weeks now, so it's becoming quite a blocker
for some elements of my current project.

Thanks.