# Way to buffer logs in logstash

**URL:** <https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334>\
**Category:** Logstash\
**Created:** [April 11, 2019, 7:03am UTC](https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334 "2019-04-11T07:03:17Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mudit\_Mehta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mudit_mehta/32/43892_2.png) [@Mudit\_Mehta](https://discuss.elastic.co/u/Mudit_Mehta)\
**Post date:** [April 11, 2019, 7:03am UTC](https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334/1 "2019-04-11T07:03:17Z")

</div>

I have configured logstash on multiple AWS instances. These logstash instances receive thousand of logs and I want a way to wait for the entire trace of logs in logstash and take action on the batch instead of a single log event. I thought of using persistent queues to buffer these logs in logstash but since I am having multiple instances the queue will not be a good option.  
Is there some way to buffer logs for some time in logstash and after that send the batches of logs that I want to send to Elastic search?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 11, 2019, 10:30am UTC](https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334/2 "2019-04-11T10:30:23Z")

</div>

> [@Mudit\_Mehta](#):
>
> I want a way to wait for the entire trace of logs in logstash

You mean like a stack trace?  
What is sending the logs to Logstash?

---

<div class="post-metadata">

**Author:** ![Mudit\_Mehta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mudit_mehta/32/43892_2.png) [@Mudit\_Mehta](https://discuss.elastic.co/u/Mudit_Mehta)\
**Post date:** [April 11, 2019, 11:48am UTC](https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334/3 "2019-04-11T11:48:12Z")

</div>

Logstash forwarder is sending the logs to logstash. I want a way to buffer logs in logstash for some time and do batch checking on them. If entire batch satisfies my criterion then i need to pass entire batch of logs to elastic search otherwise drop the entire batch.

---

<div class="post-metadata">

**Author:** ![Mudit\_Mehta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mudit_mehta/32/43892_2.png) [@Mudit\_Mehta](https://discuss.elastic.co/u/Mudit_Mehta)\
**Post date:** [April 11, 2019, 1:13pm UTC](https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334/4 "2019-04-11T13:13:35Z")

</div>

@warkolm hey can u help me regarding this ?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 11, 2019, 9:02pm UTC](https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334/5 "2019-04-11T21:02:26Z")

</div>

> [@Mudit\_Mehta](#):
>
> Logstash forwarder

This is no longer supported, you should upgrade to Beats urgently.

> [@Mudit\_Mehta](#):
>
> batch checking on them

Logstash is not really made for this, I am not sure what you can do.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 9, 2019, 9:02pm UTC](https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334/6 "2019-05-09T21:02:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.