Response from logstash to pause filebeat?


#1

I have a bunch of filebeat servers pointing to the logstash cluster via a service discovery endpoint (via consul). When I bounce the elasticsearch cluster, I usually start by stopping the filebeats, then the logstash agents, then elasticsearch; then I start elasticsearch, logstash, filebeat.

I won't be able to script the shutdown of filebeat (via service shutdown filebeat), but nevertheless I want to be able to pause it's prospecting while I bring the logstash/elasticsearch down and up.

Is there a way for me to have logstash send a "pause" response to filebeat? I don't want any data to be lost.

Thanks,
Gavin.


(Steffen Siering) #2

This kind of signal does not exist.

Beats will automatically try to reconnect if the remote becomes unavailable. As you use consul the Logstash hostnames won't change, so you are good to go. When using logstash set output.logstash.max_retries: -1. This enables infinite retry for all events. Filebeat/Winlogbeat already use infinite retry by default, no matter the max_retries setting (can not be disabled).

Why filebeat is probing for logstash to become available, the internal queues will fill up and create back-pressure. No data is collected until logstash becomes available again.


(system) #3

This topic was automatically closed after 21 days. New replies are no longer allowed.