# AWS S3 logs --\> SQS --\> ELastic search **URL:** \ **Category:** Elasticsearch\ **Created:** [May 5, 2024, 12:21am UTC](https://discuss.elastic.co/t/aws-s3-logs-sqs-elastic-search/358749 "2024-05-05T00:21:41Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Niel.devops](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@Niel.devops](https://discuss.elastic.co/u/Niel.devops)\ **Post date:** [May 5, 2024, 12:21am UTC](https://discuss.elastic.co/t/aws-s3-logs-sqs-elastic-search/358749/1 "2024-05-05T00:21:41Z")
Hi Guys, we can see that the performance of sending logs from S3 --\> Elastic SIEM is low compared to SQS. Elastic is doing enhancements in uploading data from S3 --\> SIEM directly. However, I created a Lambda function subscribed to the S3 bucket to pull and post the files on SQS. However, it's still not working as expected any Anyone who has done and tested it, please help 🙂 ![Uplaod](https://us1.discourse-cdn.com/elastic/original/3X/2/9/2942d23a174214df92adde684eaba6bbe4ed67d8.png) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 5, 2024, 4:46am UTC](https://discuss.elastic.co/t/aws-s3-logs-sqs-elastic-search/358749/2 "2024-05-05T04:46:28Z")
Hello and welcome, It is not sure what is your issue here and how it relates to any tool with Elastic stack. > [@Niel.devops](#): > > we can see that the performance of sending logs from S3 --\> Elastic SIEM is low compared to SQS Also, not sure what you mean here, peformance of what tool? How are you sending logs from S3 to Elastic? Please provide more context of what you are trying to do with some Elastic tool and what is not working. ---
**Author:** ![Niel.devops](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@Niel.devops](https://discuss.elastic.co/u/Niel.devops)\ **Post date:** [May 5, 2024, 11:47am UTC](https://discuss.elastic.co/t/aws-s3-logs-sqs-elastic-search/358749/3 "2024-05-05T11:47:48Z")
Elastic is working on the performance and reliability of the aws-s3 in polling mode.\* The aws-s3 input works in two modes: polling and SQS. In polling mode, the aws-s3 input polls a file directly from an S3 bucket (S3 \> input). In SQS mode, the S3 sends a notification to SQS, and the aws-s3 processes it (S3 \> SQS \> input)\*. However, if the S3 bucket contains a large number of S3 objects, the SQS mode offers lower latency and higher scalability. I am facing challenges with option 1 ( polling mode )\*\* and switching to option 2 ( SQS), written functions to do so in AWS. However, there is very limited documentation from Elastic on the option2 ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 5, 2024, 2:58pm UTC](https://discuss.elastic.co/t/aws-s3-logs-sqs-elastic-search/358749/4 "2024-05-05T14:58:11Z")
> [@Niel.devops](#): > > Elastic is working on the performance and reliability of the aws-s3 in polling mode.\* Can you provide a link to this information? > [@Niel.devops](#): > > However, there is very limited documentation from Elastic on the option2 It is still not clear what is your issue and how this relates to any Elastic tool. Since you are talking about the `aws-s3` input I'm assuming that you are talking about `filebeat` or `elastic agent`. To get data from s3 buckets using the `aws-s3` input it is recommended to use a SQS queue notifications instead of pooling the bucket, pooling the bucket is bad approach and I don't think that this can be improved in any significant way because pooling the bucket does not scale, so the recommendation is to use an SQS queue that receives notifications about the files in the bucket. The way this work is the following: - You configure on AWS to send a notification to an SQS queue every time a new file is created in the s3 bucket you want to consume. - Filebeat or Elastic Agent will subscribe to the SQS queue and receive a notification every time a new file is added on the S3 bucket. - In the SQS notification there is information about the file that was added, filebeat or elastic agent will then download the file from the S3 bucket and process it. The documentation on how this works on filebeat is [here](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#filebeat-input-aws-s3). How you will create the SQS queue has nothing to do with Elastic, this is done entirely on AWS side, not sure what you are trying to do with that lambda function as S3 can send the notifications to SQS natively as described [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html). The SQS notification has information about the files in the s3 bucket, not the content of the files, this way you can scale and have multiple filebeat/elastic agent subscribing to the same queue to know which files need to be downloaded and processed.