Filebeat S3 issue

[cbeprem (Prem Kumar)]

Hi team ,

I am trying to ingest logs from S3 using filebeat and SQS. It works fine when the format is

logss/source/year2020/month04/day10/hour08/2020-04-10.log

but the original format for some logs is as below and can't change or rename the folder , and it is printing s3 file not found on filebeat

logs/source/year=2020/month=04/day=10/hour=08/2020-04-10.log

can anyone suggest me how to ingest data using filebeat , when the S3 bucket has = in prefix.

filebeat.yml

output.elasticsearch:
  hosts: ["http://:9200"]
  username: "elastic"
  password: ""
setup.kibana:
  host: "http://:5601"

aws.yml

  - module: aws
  s3access:
    enabled: true
    var.queue_url: https://sqs.us-east-1.amazonaws.com/23333333333/testelklogs 

Regards,
Prem.

[Kaiyan_Sheng]

Hi @cbeprem, thanks for posting here. Could you share the debug level Filebeat log here please? Thanks! In order to turn on debug level logs, filebeat.yml needs to be changed to

logging.level: debug
output.elasticsearch:
  hosts: ["http://:9200"]
  username: "elastic"
  password: ""
setup.kibana:
  host: "http://:5601"

I will try to reproduce this error on my AWS account as well.

[cbeprem (Prem Kumar)]

The below is logs after changing the log to debug.

WARN [s3] s3/input.go:419 Cannot find s3 file
2020-05-02T17:18:39.246Z DEBUG [s3] s3/input.go:266 handleS3Objects succeed
2020-05-02T17:18:39.246Z DEBUG [s3] s3/input.go:287 Deleting message from SQS: 0xc0007ddab0
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:237 Processing 10 messages
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.277Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.278Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.278Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.279Z DEBUG [s3] s3/input.go:257 handleSQSMessage succeed and returned 1 sets of S3 log info
2020-05-02T17:18:39.289Z WARN [s3] s3/input.go:419 Cannot find s3 file
2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:266 handleS3Objects succeed
2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:287 Deleting message from SQS: 0xc0007b88a0
2020-05-02T17:18:39.289Z WARN [s3] s3/input.go:419 Cannot find s3 file
2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:266 handleS3Objects succeed
2020-05-02T17:18:39.289Z DEBUG [s3] s3/input.go:287 Deleting message from SQS: 0xc0007b86b0

this is when i tried to cancel the

s3/input.go:386 createEventsFromS3Info failed for logs/worker/source%3Dindex/year%3D2020/month%3D04/day%3D27/hour%3D14/worker_2505-simple-2020-04-27.log.log: GetObject request canceled: RequestCanceled: request context canceled

actually

the S3 prefix is
log/worker/source=index/year=2020/month=04/day=27/hour=14/worker_2505-simple-2020-04-27.log.log
but it is printing %3D for = .

Kindly help me .

Regards,
Prem.

[cbeprem (Prem Kumar)]

any updates

[Kaiyan_Sheng]

Hi @cbeprem sorry for the late response. I was able to reproduce this problem in my AWS account and here is the fix: https://github.com/elastic/beats/pull/18370 Please feel free to test this PR or wait till the later release to include this bug fix.

Right now the workaround would be to avoid "=" in the folder name in S3 It will definitely avoid the encoding issue. Thank you!