Filebeat AWS Module unable to process Logs from S3

Elastic Stack Beats
,
1

ELK-Stack: 7.11.2 and Filebeat: 7.11.2
Up until today, Filebeat-AWS Module was able to read messages(vpc logs) from sqs queue. But, of of no-where, I see filebeat is unable to process the vpc-logs. I’m seeing the below error in the filebeat logs


{"log.level":"error","@timestamp":"2021-06-10T19:16:49.147Z","log.logger":"input.s3","log.origin":{"file.name":"s3/collector.go","file.line":389},"message":"readStringAndTrimDelimiter failed: context deadline exceeded","queue_url":"SQS-Queue_URL","region":"us-east-1","ecs.version":"1.6.0"}


{"log.level":"error","@timestamp":"2021-06-10T19:16:49.148Z","log.logger":"input.s3","log.origin":{"file.name":"s3/collector.go","file.line":304},"message":"createEventsFromS3Info failed processing file from s3 bucket \"BUCKET-NAME\" with name \"AWSLogs/xxxxxx/vpcflowlogs/us-east-1/2021/06/10/xxxxxxx_vpcflowlogs_us-east-1_fl-0f728d964f22ff25f_20210610T1345Z_ea3fd52f.log.gz\": context deadline exceeded","queue_url":"SQS-Queue_URL","region":"us-east-1","ecs.version":"1.6.0"}
2

On S3 bucket the Metadata is: Content-Type: application/x-gzip

3

Filebeat AWS Config is as below:


- module: aws
  vpcflow:
    enabled: true
    var.queue_url: SQS-Queue_URL
    var.credential_profile_name: xxxx
    var.api_timeout: 120s
    var.endpoint: amazonaws.com
    var.role_arn: <Role-ARN>
4

@legoguy1000 @Kaiyan_Sheng Can you please review & suggest on the issue above.
Any help/review would be great!

5

Unfortunately that one IDK. Maybe the issue is the Content-Type: application/x-gzip

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.