Filebeat Stops Processing logs for AWS module

Elastic Stack Beats
1

I am facing exactly the same issue as this thread Filebeat stops proccessing s3 input with no error.

Also, I experienced sometimes Filebeat suddenly stops processing data for other modules too, I faced it with Gsuite module as well. However, it shows no error while log.level set to debug

here are my logs :

Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.938Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.938Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.938Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.939Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.939Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.939Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.939Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.940Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.941Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.942Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.946Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.949Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.949Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.950Z        WARN        [input.s3]        s3/collector.go:189        Half of the set visibilityTimeout passed, visibility timeout needs to be updated        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:46 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:46.976Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.015Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.015Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.016Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.016Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.022Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.022Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.022Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.022Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.025Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.026Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.031Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.031Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.032Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}
Jun 15 11:56:47 ip-172-31-26-170 filebeat[368069]: 2021-06-15T11:56:47.034Z        INFO        [input.s3]        s3/collector.go:196        Message visibility timeout updated to 300 seconds        {"queue_url": "https://my_queeue_url", "region": "ap-south-1"}

is there any workaround for this? Need help ASAP, facing this in the production environment :neutral_face:

2

Hi!

What version do you use? Also could you provide full configuration for reference?

@Kaiyan_Sheng could you please help with this one?

1 Like
3

Hey @ChrsMark Thanks for the quick response. I have resolved it by restarting the server where Filebeat is running. I am suspecting its issue with high CPU usage.

4

Hi! What version of Filebeat are you running? There was a major bug in Filebeat aws module and was fixed by [Filebeat] aws-s3 input visibility timeout extensions lead to deadlocked ACKHandlers · Issue #25750 · elastic/beats · GitHub and backported to 7.13. If it's possible, could you give 7.13.2 a try, thanks!!

5

Hey @Kaiyan_Sheng, Thanks for the response. I am using 7.10.1

1 Like
6

OK! Could you give 7.13.2 a try?

7

Would it be compatible with Elasticsearch and Kibana 7.10

8

It should be based on the compatibility matrix :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.