publisher#Client.Publish and backpressure

[HeyImAlex (Alex Guerra)]

I'm trying to do some work on journalbeat, but the specifics around how publisher#Client.Publish works are kinda murky to me.

Just for context, journalbeat basically polls journald for new entries, so the "event source" can be modeled in a straight forward way as a channel. A goroutine reads from this channel and then calls Client.Publish with a signaller.

My question is do I need to implement backpressure myself? It seems like because Publish is async, a flood of events on the event channel (ie at startup time) will cause an explosion of in-flight requests. Since we're just polling from journald, backpressure is easy to implement on the producer side (just don't pull the next value until the consumer is ready), but the Client has no way to indicate I need to slow down. I imagine the issues here are similar to those handled by filebeat, and I tried to take a look at it but it's really complex and looks like it's not using the libbeat Publisher anyway.

Any suggestions here?

[andrewkroh]

If you use the Sync and Gauranteed options in the PublishEvents call, it will block until that batch of events is completely published. This is how winlogbeat is implemented. https://github.com/elastic/beats/blob/master/winlogbeat/beater/winlogbeat.go#L209

[HeyImAlex (Alex Guerra)]

Ok, thanks, I'll look more closely into how winlogbeat is implemented. I was mostly wondering if libbeat handled this itself or if it was on me to not flood it with events.