Safely publish an event using libbeat

Elastic Stack Beats
3

options understood by Publish:

  • publisher.Sync: block PublishEvent until either event has been send or event is dropped due to max_retries
  • publisher.Guaranteed: Never drop event. Keep event in output pipeline until ACKed by output. Can be used without sync for async guaranteed publishing.
  • publisher.Signal: Callback for publish requests. E.g. used in filebeat in conjunction with publisher.Guaranteed to update file state offsets once events have been ACKed by output.

Filebeat supports both mode. Sync publishing: https://github.com/elastic/beats/blob/master/filebeat/beater/publish.go#L110
and async publishing: https://github.com/elastic/beats/blob/master/filebeat/beater/publish.go#L166

If output (logstash/elasticsearch/...) is unresponsive and buffers in publisher pipeline (both, in sync and async mode) are full, PublishEvent(s) will block. In order to guarantee proper shutdown one has to dicsonnect from publisher pipeline via Close. Once Close is called, the publisher pipeline will not give any guarantees to events being queued. Events might be send or not.

1 Like