Brand new to beats and really loving it so far. Just getting starting with optimizations and am not seeing anything for any type of event caching. I see the parameters for retries on publishing events:
# The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat, ignore the max_retries setting and retry until # all events are published. Set max_retries to a value less than 0 to retry # until all events are published. The default is 3. #max_retries: 3
Hi @BrandonF, what do you mean by event caching? What are you trying to optimize?
A batch of events is read from the event log, stored in memory, sent to the output (it retries indefinitely until they delivered), then it repeats the cycle.
We wanted to check while processing the events (read from event log-->store in memory) when my output link is not reached, where are the events which are read get stored?
While processing is there anything written on to local disk and its path
The registry file is updated after a batch of events has been successfully delivered to the output. The registry stores the last successfully delivered record_number for each event log.
With our previous connector architecture we were able to configure files (anywhere from 500MB to 10GB) that were held on disk that contained events when the destination was down/unreachable. Essentially giving us more time to recover without having to drop events, even if the Windows Event Logs started to roll over.
Doesn't sound like this is an option with beats, but the closest configuration is to just set max_retries to -1.
The event logs are files on disk so they are basically a buffer. Whether Winlogbeat stores the events in a persistent queue on disk or the data is stored in the event logs doesn't make a huge difference. The amount of down time you can withstand without losing data is still limited by the amount of local diskspace available. So if you want to have more buffer to account for down time of an output you can increase the size of your event logs.
We plan on add a disk-backed queue option to Beats sometime in the 6.x releases. So you will have this choice in the future. We want this for data sources that are not already on disk (like metrics).
There is a persistent queue option in Logstash if you want to have a buffer between WLB and ES that you can rely on when ES is unavailable.
Really good info...Thank you! I had considered increasing the event logs in this situation. Looking forward to seeing what's to come in the new releases.
Unless you are actually setting the value in your config file the behavior will be "guaranteed delivery" where by it will continuously retry until success. The config file and docs could be more clear on this.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
