Safely publish an event using libbeat

Elastic Stack Beats
1

So I'm at a loss on how to go about publishing an event and being sure it was a success. If I use publisher.Client.PublishEvent(m, publisher.Sync) and the output (in my test case logstash) isn't up, it blocks forever instead of timing out and returing false. This hangs the beat requiring a SIGKILL to stop it. Am I doing something wrong or is this a bug?

2

The client has a Close() method which must be used in the case of sync publisher. See here the implementation in Filebeat: https://github.com/elastic/beats/blob/master/filebeat/beater/publish.go#L125

Here are the docs for the client: https://godoc.org/github.com/elastic/beats/libbeat/publisher#Client

3

options understood by Publish:

  • publisher.Sync: block PublishEvent until either event has been send or event is dropped due to max_retries
  • publisher.Guaranteed: Never drop event. Keep event in output pipeline until ACKed by output. Can be used without sync for async guaranteed publishing.
  • publisher.Signal: Callback for publish requests. E.g. used in filebeat in conjunction with publisher.Guaranteed to update file state offsets once events have been ACKed by output.

Filebeat supports both mode. Sync publishing: https://github.com/elastic/beats/blob/master/filebeat/beater/publish.go#L110
and async publishing: https://github.com/elastic/beats/blob/master/filebeat/beater/publish.go#L166

If output (logstash/elasticsearch/...) is unresponsive and buffers in publisher pipeline (both, in sync and async mode) are full, PublishEvent(s) will block. In order to guarantee proper shutdown one has to dicsonnect from publisher pipeline via Close. Once Close is called, the publisher pipeline will not give any guarantees to events being queued. Events might be send or not.

1 Like
4

This topic was automatically closed after 21 days. New replies are no longer allowed.