Elastic Cloudtrail for Cognito integration

Elastic Stack Elastic Agent
1

Hi

I am writing logs from my AWS Cognito service to an S3 bucket

I want to read those logs and ingest them in an Elastic Cloud instance using the AWS integration

Cognito generats logs in an S3 bucket with the following pattern...

Amazon S3 > Buckets > data-access-cloudtrail-logs > AWSLogs > 123456789 >
CloudTrail > ap-south-1 > 2024 > 10 > 15 > 15

There is a new folder for each day and each final folder has a set of .gz files

I set up an agent and specified first the ap-south-1 bucket but it didn't pick anything up

Then I specified the final 15 bucket, just to point it to the directory with the actual .gz log files and it still doesn't ingest them...

First question is:

  1. How do I debug if the agent read the logs or not - I can see level4 logs in the agent logs section but I dont' see an error for access
  2. What's the correct S3 path for Cloudtrail access? Since it will create a new subdirectory in the bucket every day

Thanks

2

Hey @sami-obvio,

If I understand the pattern, you have AWS S3 bucket named data-access-cloudtrail-logs and there are folders and subfolders underneath it. You would like to read the logs from specific subfolder, say ap-south-1.

In this case, you will want to first configure Bucket ARN setting in the integration with the ARN of the S3 bucket. It could be something like this: arn:aws:s3:::data-access-cloudtrail-logs. You can find your bucket ARN from AWS console.

Then, if you want to read from certain folder (prefix), then you should enter this folder path into Bucket List Prefix integration setting with a value AWSLogs/123456789/CloudTrail/ap-south-1 assuming thats the correct folder path in your S3 bucket. This integration setting can be found sometimes in Advanced options.

For the question regarding debugging, you can change log level to DEBUG and also extract the agent diagnostics to view the logs.

More details on enabling debug logs and collecting diagnostics here:
Standalone Agent DEBUG
Standalone Agent diagnostics
Fleet Agent DEBUG
Fleet Agent diagnostics

1 Like
3

Thank you Krishna

There were a couple of issues that I discovered...

Firstly, I was using a Cognito user which expired... so I switched to an IAM user that had no expiry

Secondly, I mixed up the S3 path with the ARN.. I will try this again and update here