Is there a buffer in Logstash to keep data in case of Elasticsearch (or any other output-taget) is down? If so, can I configure the size etc.? So I can estimate for how long we can accept elasticsearch is down.
Logstash currently doesn't have such a buffer but I think it's planned for 2.0. For now I suggest you use a broker like Redis or RabbitMQ as a middle man.
What kind of inputs do you have?
okay I see. It is logdata, so I could do a manual operation. What about the logstash-forwarder, does that have a buffer? Or does it just keep track of where it left off if the logstash-server is not responding?
Logstash's file input as well as logstash-forwarder will sense a clogged output and will back off by simply stop reading from the input files. However, if there's an outage while a logfile is rotated I'm pretty sure at least Logstash won't handle the situation gracefully (meaning old messages not already read from the file rolled to a new filename won't be read but new messages from the possibly new file will be read).
Then is there a chance for data loss if Elasticsearch is down and Logstash (1.5) already read the data from log file and is ready to send. How is this handled?
Logstash won't update sincedb with the new file offset until after it has posted the events to Logstash's internal 20-item queue so that's the maximum amount of messages you might lose (ignoring the previsouly describe file rotation problem).