# Filebeat S3 issue **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 29, 2020, 9:19pm UTC](https://discuss.elastic.co/t/filebeat-s3-issue/230441 "2020-04-29T21:19:37Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![cbeprem](https://avatars.discourse-cdn.com/v4/letter/c/51bf81/32.png) [@cbeprem](https://discuss.elastic.co/u/cbeprem)\ **Post date:** [April 29, 2020, 9:19pm UTC](https://discuss.elastic.co/t/filebeat-s3-issue/230441/1 "2020-04-29T21:19:38Z")
Hi team , I am trying to ingest logs from S3 using filebeat and SQS. It works fine when the format is logss/source/year2020/month04/day10/hour08/2020-04-10.log but the original format for some logs is as below and can't change or rename the folder , and it is printing s3 file not found on filebeat logs/source/year=2020/month=04/day=10/hour=08/2020-04-10.log can anyone suggest me how to ingest data using filebeat , when the S3 bucket has = in prefix. **filebeat.yml** ```auto output.elasticsearch: hosts: ["http://:9200"] username: "elastic" password: "" setup.kibana: host: "http://:5601" ``` **aws.yml** ```auto - module: aws s3access: enabled: true var.queue_url: https://sqs.us-east-1.amazonaws.com/23333333333/testelklogs ``` Regards, Prem. ---
**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\ **Post date:** [April 30, 2020, 6:23pm UTC](https://discuss.elastic.co/t/filebeat-s3-issue/230441/2 "2020-04-30T18:23:54Z")
Hi @cbeprem, thanks for posting here. Could you share the debug level Filebeat log here please? Thanks! In order to turn on debug level logs, `filebeat.yml` needs to be changed to ```auto logging.level: debug output.elasticsearch: hosts: ["http://:9200"] username: "elastic" password: "" setup.kibana: host: "http://:5601" ``` I will try to reproduce this error on my AWS account as well. ---
**Author:** ![cbeprem](https://avatars.discourse-cdn.com/v4/letter/c/51bf81/32.png) [@cbeprem](https://discuss.elastic.co/u/cbeprem)\ **Post date:** [May 2, 2020, 5:51pm UTC](https://discuss.elastic.co/t/filebeat-s3-issue/230441/3 "2020-05-02T17:51:15Z")
The below is logs after changing the log to debug. WARN [s3] s3/input.go:419 Cannot find s3 file 2020-05-02T17:18:39.246Z DEBUG [s3] s3/input.go:266 handleS3Objects succeed 2020-05-02T17:18:39.246Z DEBUG [s3] s3/input.go:287 Deleting message from SQS: 0xc0007ddab0 2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:237 Processing 10 messages 2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.278Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.278Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.279Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info 2020-05-02T17:18:39.289Z WARN [s3] s3/input.go:419 Cannot find s3 file 2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:266 handleS3Objects succeed 2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:287 Deleting message from SQS: 0xc0007b88a0 2020-05-02T17:18:39.289Z WARN [s3] s3/input.go:419 Cannot find s3 file 2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:266 handleS3Objects succeed 2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:287 Deleting message from SQS: 0xc0007b86b0 this is when i tried to cancel the s3/input.go:386 createEventsFromS3Info failed for logs/worker/source%3Dindex/year%3D2020/month%3D04/day%3D27/hour%3D14/worker\_2505-simple-2020-04-27.log.log: GetObject request canceled: RequestCanceled: request context canceled actually the S3 prefix is log/worker/source=index/year=2020/month=04/day=27/hour=14/worker\_2505-simple-2020-04-27.log.log but it is printing %3D for = . Kindly help me . Regards, Prem. ---
**Author:** ![cbeprem](https://avatars.discourse-cdn.com/v4/letter/c/51bf81/32.png) [@cbeprem](https://discuss.elastic.co/u/cbeprem)\ **Post date:** [May 6, 2020, 5:42pm UTC](https://discuss.elastic.co/t/filebeat-s3-issue/230441/4 "2020-05-06T17:42:23Z")
any updates ---
**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\ **Post date:** [May 7, 2020, 9:53pm UTC](https://discuss.elastic.co/t/filebeat-s3-issue/230441/5 "2020-05-07T21:53:49Z")
Hi @cbeprem sorry for the late response. I was able to reproduce this problem in my AWS account and here is the fix: [https://github.com/elastic/beats/pull/18370](https://github.com/elastic/beats/pull/18370) Please feel free to test this PR or wait till the later release to include this bug fix. Right now the workaround would be to avoid "=" in the folder name in S3 🤔 It will definitely avoid the encoding issue. Thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 4, 2020, 9:53pm UTC](https://discuss.elastic.co/t/filebeat-s3-issue/230441/6 "2020-06-04T21:53:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.