How to receive ack of publish in 6.0 libbeat

[JamesB]

Hi,

I've just started looking at creating a custom beat using the 6.0 branch. I want to guarantee delivery of events and also be notified when an event has been published. What is the best way to do this?

I've found the ClientConfig which can be passed into Publisher.ConnectWith and the option GuaranteedSend seems the correct one. When looking at acknowledgements there is a property ACKEventswhich takes a function but the documentation says:

ACKEvents reports the events private data of recently acknowledged events.

What does this mean? I've tried passing in a function to this and printing out the contents received but they are empty. I've noticed the Private property on Event but the code says // for beats private use. Can I use the Private property to track ids for events?

Is there an existing beat that I could look at that deals with acknowledging success to the system being read from?

[steffens (Steffen Siering)]

Hi,

which callback you want to use somehow depends on your Beat and how you want to store the last state in order to continue sending. The ACKEvents handler is somewhat correct, but in the new publisher pipeline in 6.0 a total of 4 (slightly different) ACK callbacks do exists.

The doc comment on Private is maybe a little misleading. for beats private usage means your own beat, not libbeat internal. Libbeat will only process Timestamp, Fields and Meta. The contents in Private is asynchronously passed to the ACK handler.

All ACK handlers report acked events in the same order events have been published to the pipeline. Even if events are ACKed out of order (e.g. due to load-balancing).

First of all you have to choose if you want the ACK handling local to the beat.Client instance (each worker acquiring new events should have it's own instance) or globally. If the beat.Client is closed, you will receive no more ACKs, even if events are currently processed and ackeded by an output. You can still mix this using the WaitClose setting to block on the beat.Client close operation until all events published have been either acked or the timedout occurs.

For local ACK handling you have to configure the ACK handler via ClientConfig. For global ACK handling use SetACKHandler on the Publisher being passed to your beat. This must be done before the first call to Connect or ConnectWith.

Beats outputs process events in batches. The event acks are batched up into fewer calls to the ACK handlers (which are called asynchronously to the publisher). If the state to be persistent is some kind of sequential (ever increasing) ID (e.g. timestamp, counter, offset), you might not be interested in processing all events ever published. In this case use the ACKLastEvent(s) callback. This will report the Private field of the most recent published and ACKed event. The global callback reports an array of Private fields, each entry representing another beat.Client instance whose most recent event has been ACKed.

In beats we currently use the global ACK handlers only (filebeat and winlogbeat). These are still active when all beat.Client instances have been closed. As winlogbeat is somewhat simpler, rather have a look at winlogbeat:

• When creating the beat.Event winlogbeat sets the Private field to the next checkpoint value.
• the ACK handler finally persists the states already being ACKed
• also note how winlogbeat uses the local counting ACK handler to report events being ACKed for particular workers

Filebeat uses the same interfaces as Winlogbeat.

Currently filebeat and winlogbeat have different ways of persisting and reading the state (also gc'ing old state not required anymore). For the future we hope to introduce unified interfaces in libbeat, so to help with implementing logic for storing, reloading, and gc'ing checkpoints.

As you want to pass the ACK back to the source system, you might be interested in the local ACK handlers. Depending on how ack is reported back to the source system, you might be even good using the ACKCount handler only.

[JamesB]

Thank you, that's really helpful. I'll have a look at those options and Winlogbeat and try them out.