Filebeat crashed when processing messages from sqs queue

Elastic Stack Beats
1

Hi:

I am running filebeat 7.15.0 to process S3 object creation event messages from AWS SQS. The queue is configured to receive AWS ELB log files. After a few minutes, filebeat 7.15.0 crashed with the following error.

The same program used to work fine with filebeat 7.14.1. This crash happened after I upgraded filebeat from 7.14.1 to 7.15.0

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x55859828e20b]

goroutine 168 [running]:
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).download(0xc000dbe000, 0x82ef4a26, 0x55859b535100, 0x0, 0xc000d99b08, 0x1, 0x1, 0x0)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/s3_objects.go:176 +0x10b
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object(0xc000dbe000, 0x0, 0x0)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/s3_objects.go:136 +0x178
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsS3EventProcessor).processS3Events(0xc000139e50, 0x558599c5e778, 0xc000d34140, 0xc000d8e020, 0xc0010a4d80, 0x405, 0x0, 0x0)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs_s3_event.go:245 +0x469
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsS3EventProcessor).ProcessSQS(0xc000139e50, 0x558599c5e778, 0xc000d34140, 0xc000d8c000, 0x0, 0x0)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs_s3_event.go:111 +0x23a
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsReader).Receive.func1(0xc000d34300, 0xc00058f2f0, 0x558599c5e778, 0xc000d34140, 0xc000ac1e00, 0xc000ab6560, 0xc000ab6590, 0x0, 0x0, 0xc000ab65c0, ...)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs.go:90 +0x142
created by github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsReader).Receive
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs.go:82 +0x4b5
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x55859828e20b]

goroutine 167 [running]:
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).download(0xc0010b6000, 0x82eeff9e, 0x55859b535100, 0x0, 0xc000f83b08, 0x1, 0x1, 0x0)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/s3_objects.go:176 +0x10b
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object(0xc0010b6000, 0x0, 0x0)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/s3_objects.go:136 +0x178
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsS3EventProcessor).processS3Events(0xc000139e50, 0x558599c5e778, 0xc000d34140, 0xc000ab6650, 0xc0010a4480, 0x40a, 0x0, 0x0)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs_s3_event.go:245 +0x469
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsS3EventProcessor).ProcessSQS(0xc000139e50, 0x558599c5e778, 0xc000d34140, 0xc0010a6980, 0x0, 0x0)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs_s3_event.go:111 +0x23a
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsReader).Receive.func1(0xc000d34300, 0xc00058f2f0, 0x558599c5e778, 0xc000d34140, 0xc000ac1dd0, 0xc000ab6480, 0xc000ab64b0, 0x0, 0x0, 0xc000ab64e0, ...)
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs.go:90 +0x142
created by github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsReader).Receive
	/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs.go:82 +0x4b5
2

I was able to fix the issue myself. The documentation of elastic on k8s is wrong.
The service account for fleet 7.15.0 must allow resource list permission of nodes and pods.
This service account permission is different from fleet 7.14.1

rules:

  • apiGroups: [""] # "" indicates the core API group
    resources:
    • pods
    • nodes
3

I am getting the same error as mentioned above.
But i have setup the filebeat as docker container using composer.
Same configuration works fine for filebeat:7.14.2 version and crashes for 7.15.0 version.

Even the Fleet agent fails when ELB logs integration is added saying "Filebeat has entered Degraded state"

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.