Hi @cbeprem, thanks for posting here. Could you share the debug level Filebeat log here please? Thanks! In order to turn on debug level logs, filebeat.yml needs to be changed to
The below is logs after changing the log to debug.
WARN [s3] s3/input.go:419 Cannot find s3 file
2020-05-02T17:18:39.246Z DEBUG [s3] s3/input.go:266 handleS3Objects succeed
2020-05-02T17:18:39.246Z DEBUG [s3] s3/input.go:287 Deleting message from SQS: 0xc0007ddab0
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:237 Processing 10 messages
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.278Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.278Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.279Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.289Z WARN [s3] s3/input.go:419 Cannot find s3 file
2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:266 handleS3Objects succeed
2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:287 Deleting message from SQS: 0xc0007b88a0
2020-05-02T17:18:39.289Z WARN [s3] s3/input.go:419 Cannot find s3 file
2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:266 handleS3Objects succeed
2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:287 Deleting message from SQS: 0xc0007b86b0
Hi @cbeprem sorry for the late response. I was able to reproduce this problem in my AWS account and here is the fix: https://github.com/elastic/beats/pull/18370 Please feel free to test this PR or wait till the later release to include this bug fix.
Right now the workaround would be to avoid "=" in the folder name in S3 It will definitely avoid the encoding issue. Thank you!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.