Logstash s3 input with iam role

[niraj_kumar]

Hi ,

I am trying to use logstash s3 input plugin to process cloudtrail data. My configuration is all OK but as i am running this logstash on prem, my access to s3 bucket is via a IAM role. I cannot get access and secret keys as my org. does not allow the use of that.

Is there a way anyone knows about to process this data using an IAM role.

I have configure my CLI and i can access things via command line.

--
Niraj

[Tito (Lukasz)]

Hello,

I also granted S3 bucket access via IAM role. My logstash input looks like:

input
{
s3
{
bucket => "name_of_bucket"
prefix => "logs/"
interval => 60
region => "ap-southeast-2"
type => "input_type"
}
}

and it's working, so without providing access and secret keys.

regards,
Lukasz

[Tito (Lukasz)]

Hello,

and the IAM role looks like:

Show Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "---",
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::name_of_the_bucket"]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": ["arn:aws:s3:::name_of_the_bucket/*"]
}
]
}

regards,
Lukasz

[niraj_kumar]

@Tito What you are running is on an EC2 instance i believe. I am on-prem instances where i cannot attach IAM roles. I wrote a self script to flush environment variables. But was just curious if there is a way or trick out there to help achieve this scenario.

--
Niraj

[niraj_kumar]

Seems like if you have set an ENV variable and try to reflush it using a script while the logstash is ingesting. Logstash will never find this change and will stop ingesting because the credentials expires after an hour. I am kind of blocked as my IT does not ever allow the usage of access keys.

--
Niraj