nitzang1
(Nitzan Gotlib)
January 5, 2022, 7:59pm
1
Hi all,
I'm facing an issue where filebeat pubsub input stop pulling events just after throwing this error:
2022-01-05T15:30:24.404Z ERROR [gcp.pubsub] gcppubsub/input.go:141 context deadline exceeded {"pubsub_project": "project-1", "pubsub_topic": "some-logs", "pubsub_subscription": {"Name":"some-logs","NumGoroutines":1,"MaxOutstandingMessages":-1,"Create":false}}
restarting the service works but is not a solution.
Does anyone have any experience with that?
filebeat.yml:
filebeat.inputs:
- type: gcp-pubsub
project_id: project-1
topic: some-logs
subscription:
name: some-logs
create: false
num_goroutines: 1
max_outstanding_messages: -1
credentials_file: /etc/gcp/sa_credentials.json
Thanks
nitzang1
(Nitzan Gotlib)
January 25, 2022, 4:25pm
2
This is a known bug and was fixed on 7.17.0:
opened 07:46PM - 08 Dec 21 UTC
closed 01:16AM - 06 Jan 22 UTC
bug
Filebeat
v8.0.0
Team:Security-External Integrations
v7.17.0
The gcppubsub input sometimes stops due to a timed-out context. Our input code d… oes not use timed contexts AFAIK so I suspect the issue lies with the pubsub library. The library is supposed to automatically retry `Receive` calls as per https://pkg.go.dev/gocloud.dev/pubsub#Subscription.Receive.
So as a solution I think we can do two things:
- ✅ Upgrade the pubsub library from v1.0.1 to the latest on the hopes that this issue has been addressed.
- 🚫 Add our own retry loop with backoff that will continue the subscription.Receive call. (update: library update alone appears to be a reliable fix)
For confirmed bugs, please report:
- Version: 7.15.2
- Steps to Reproduce:
1. Configure pubsub input.
2. Let it run for a few days.
3. Input will stop.
`2021-11-09T14:37:40.604Z ERROR [gcp.pubsub] gcppubsub/input.go:141 context deadline exceeded`
`2021-11-09T14:37:40.604Z INFO [gcp.pubsub] gcppubsub/input.go:142 Pub/Sub input worker has stopped. `
Relevant code:
https://github.com/elastic/beats/blob/545598f48058408996529af3252b1b28d6dbf53d/x-pack/filebeat/input/gcppubsub/input.go#L168-L174
Thanks
system
(system)
Closed
February 22, 2022, 6:25pm
3
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.