Filebeat Stops Processing logs for AWS module

ankitdevnalkar · June 15, 2021, 12:05pm

I am facing exactly the same issue as this thread Filebeat stops proccessing s3 input with no error.

Also, I experienced sometimes Filebeat suddenly stops processing data for other modules too, I faced it with Gsuite module as well. However, it shows no error while log.level set to debug

here are my logs :

Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.938Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.938Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.938Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.939Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.939Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.939Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.939Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.940Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.941Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.942Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.946Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.949Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.949Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.976Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.015Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.015Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.016Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.016Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.022Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.022Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.022Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.022Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.025Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.031Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.031Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.032Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.034Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}

is there any workaround for this? Need help ASAP, facing this in the production environment

ChrsMark · June 15, 2021, 1:38pm

Hi!

What version do you use? Also could you provide full configuration for reference?

@Kaiyan_Sheng could you please help with this one?

ankitdevnalkar · June 15, 2021, 3:10pm

Hey @ChrsMark Thanks for the quick response. I have resolved it by restarting the server where Filebeat is running. I am suspecting its issue with high CPU usage.

Kaiyan_Sheng · June 15, 2021, 3:42pm

Hi! What version of Filebeat are you running? There was a major bug in Filebeat aws module and was fixed by [Filebeat] aws-s3 input visibility timeout extensions lead to deadlocked ACKHandlers · Issue #25750 · elastic/beats · GitHub and backported to 7.13. If it's possible, could you give 7.13.2 a try, thanks!!

ankitdevnalkar · June 15, 2021, 5:11pm

Hey @Kaiyan_Sheng, Thanks for the response. I am using 7.10.1

Kaiyan_Sheng · June 15, 2021, 5:34pm

OK! Could you give 7.13.2 a try?

ankitdevnalkar · June 15, 2021, 5:42pm

Would it be compatible with Elasticsearch and Kibana 7.10

Kaiyan_Sheng · June 15, 2021, 7:38pm

It should be based on the compatibility matrix

system · July 13, 2021, 9:39pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat stops proccessing s3 input with no error Beats filebeat	4	1154	January 20, 2020
Filebeat AWS Module unable to process Logs from S3 Beats beats-module , filebeat	5	575	July 12, 2021
Filebeat AWS Module problem Beats beats-module , filebeat	5	1420	June 16, 2020
Filebeat not ingesting logs from s3 Beats filebeat	10	1279	December 19, 2019
Filebeat and AWS Module unable to get CloudTrail logs Beats beats-module , filebeat	2	1416	June 25, 2020

Filebeat Stops Processing logs for AWS module

Related topics