# Buffer data in Logstash if Elasticsearch is down

**URL:** https://discuss.elastic.co/t/buffer-data-in-logstash-if-elasticsearch-is-down/29127
**Category:** Logstash
**Created:** [September 11, 2015, 11:03am UTC](https://discuss.elastic.co/t/buffer-data-in-logstash-if-elasticsearch-is-down/29127 "2015-09-11T11:03:06Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![scaarup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scaarup/32/4615_2.png) [@scaarup](https://discuss.elastic.co/u/scaarup)
#### Post date: [September 11, 2015, 11:03am UTC](https://discuss.elastic.co/t/buffer-data-in-logstash-if-elasticsearch-is-down/29127/1 "2015-09-11T11:03:06Z")

</div>

Is there a buffer in Logstash to keep data in case of Elasticsearch (or any other output-taget) is down? If so, can I configure the size etc.? So I can estimate for how long we can accept elasticsearch is down.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 11, 2015, 11:15am UTC](https://discuss.elastic.co/t/buffer-data-in-logstash-if-elasticsearch-is-down/29127/2 "2015-09-11T11:15:33Z")

</div>

Logstash currently doesn't have such a buffer but I think it's planned for 2.0. For now I suggest you use a broker like Redis or RabbitMQ as a middle man.

What kind of inputs do you have?

---

<div class="post-metadata">

### Author: ![scaarup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scaarup/32/4615_2.png) [@scaarup](https://discuss.elastic.co/u/scaarup)
#### Post date: [September 13, 2015, 10:32am UTC](https://discuss.elastic.co/t/buffer-data-in-logstash-if-elasticsearch-is-down/29127/3 "2015-09-13T10:32:51Z")

</div>

okay I see. It is logdata, so I could do a manual operation. What about the logstash-forwarder, does that have a buffer? Or does it just keep track of where it left off if the logstash-server is not responding?

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 13, 2015, 2:51pm UTC](https://discuss.elastic.co/t/buffer-data-in-logstash-if-elasticsearch-is-down/29127/4 "2015-09-13T14:51:46Z")

</div>

Logstash's file input as well as logstash-forwarder will sense a clogged output and will back off by simply stop reading from the input files. However, if there's an outage while a logfile is rotated I'm pretty sure at least Logstash won't handle the situation gracefully (meaning old messages not already read from the file rolled to a new filename won't be read but new messages from the possibly new file will be read).

---

<div class="post-metadata">

### Author: ![vilas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vilas/32/613_2.png) [@vilas](https://discuss.elastic.co/u/vilas)
#### Post date: [December 2, 2015, 10:01pm UTC](https://discuss.elastic.co/t/buffer-data-in-logstash-if-elasticsearch-is-down/29127/5 "2015-12-02T22:01:11Z")

</div>

Then is there a chance for data loss if Elasticsearch is down and Logstash (1.5) already read the data from log file and is ready to send. How is this handled?

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 3, 2015, 6:56am UTC](https://discuss.elastic.co/t/buffer-data-in-logstash-if-elasticsearch-is-down/29127/6 "2015-12-03T06:56:37Z")

</div>

Logstash won't update sincedb with the new file offset until after it has posted the events to Logstash's internal 20-item queue so that's the maximum amount of messages you might lose (ignoring the previsouly describe file rotation problem).

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:20am UTC](https://discuss.elastic.co/t/buffer-data-in-logstash-if-elasticsearch-is-down/29127/7 "2017-07-06T05:20:15Z")

</div>


