Buffer data in Logstash if Elasticsearch is down


#1

Is there a buffer in Logstash to keep data in case of Elasticsearch (or any other output-taget) is down? If so, can I configure the size etc.? So I can estimate for how long we can accept elasticsearch is down.


(Magnus Bäck) #2

Logstash currently doesn't have such a buffer but I think it's planned for 2.0. For now I suggest you use a broker like Redis or RabbitMQ as a middle man.

What kind of inputs do you have?


#3

okay I see. It is logdata, so I could do a manual operation. What about the logstash-forwarder, does that have a buffer? Or does it just keep track of where it left off if the logstash-server is not responding?


(Magnus Bäck) #4

Logstash's file input as well as logstash-forwarder will sense a clogged output and will back off by simply stop reading from the input files. However, if there's an outage while a logfile is rotated I'm pretty sure at least Logstash won't handle the situation gracefully (meaning old messages not already read from the file rolled to a new filename won't be read but new messages from the possibly new file will be read).


(Vilas Reddy Podduturi) #5

Then is there a chance for data loss if Elasticsearch is down and Logstash (1.5) already read the data from log file and is ready to send. How is this handled?


(Magnus Bäck) #6

Logstash won't update sincedb with the new file offset until after it has posted the events to Logstash's internal 20-item queue so that's the maximum amount of messages you might lose (ignoring the previsouly describe file rotation problem).


(system) #7