# Handle batching just like Persistent Queue using the In-Memory Queue

**URL:** <https://discuss.elastic.co/t/handle-batching-just-like-persistent-queue-using-the-in-memory-queue/383453>\
**Category:** Logstash\
**Created:** [November 14, 2025, 7:46pm UTC](https://discuss.elastic.co/t/handle-batching-just-like-persistent-queue-using-the-in-memory-queue/383453 "2025-11-14T19:46:02Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Oscar\_Moyeda](https://avatars.discourse-cdn.com/v4/letter/o/cdc98d/32.png) [@Oscar\_Moyeda](https://discuss.elastic.co/u/Oscar_Moyeda)\
**Post date:** [November 14, 2025, 7:46pm UTC](https://discuss.elastic.co/t/handle-batching-just-like-persistent-queue-using-the-in-memory-queue/383453/1 "2025-11-14T19:46:02Z")

</div>

Hi everyone. I am using **Logstash Version: 9.1.0**  
I was debugging my output plugin and I found out that when using the default In-memory queue, Logstash would batch the events into smaller chunks of events.

I have script to write **100 records per second** to a file I use as input for Logstash.  
The output plugin receives the batched events and prints the number of events per batch.  
Now here is what I found out while testing the different parameters in logstash.yml:

**Default - In-memory Queue:**  
With 100 logs per second written to file and default **logstash.yml** configuration (no modifications), the batch sizes look like this: `20, 33, 27, 20` - total: 100 events.

**Using Persistent Queue:**  
100 logs per second written to file (same as before), and the only change in **logstash.yml** is setting `queue.type: persisted`. The 100 records were batched in a single batch with a size of 100 events.

**The goal:**  
I would like to know if there is a way to have the In-memory Queue batch events like Persistent Queue does? What I want to achieve is to have less small requests, and instead have a single request or batch with all those events (until they fill `pipeline.batch.size: 100` for example).

**What I have tried:**  
I already tried playing with these pipeline parameters in Logstash.yml:

- pipeline.batch.size
- pipeline.batch.delay

I’ve read that those two are the only parameters that modify the batch behavior, and `pipeline.batch.size` only sets the max number of events per batch, but there might be something I’m missing.

Here is my logstash .conf file in case is useful. `my_plugin` only uploads the events to an endpoint and prints to stdout the number of events in each batch.

```auto
input {
   file { 
    path => "/path_to_test_logs/logs.txt" 
    start_position => "beginning" 
    sincedb_path => "/dev/null"
  }
 }

output {
  my_plugin {}
}

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [November 15, 2025, 11:03pm UTC](https://discuss.elastic.co/t/handle-batching-just-like-persistent-queue-using-the-in-memory-queue/383453/2 "2025-11-15T23:03:04Z")

</div>

Have you tried to add `pipeline.workers: 1` in logstash.yml or to set `pipeline.batch.size = 100` and increase to `pipeline.batch.delay = 250`?  
Don't forget to restart LS.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 16, 2025, 1:48pm UTC](https://discuss.elastic.co/t/handle-batching-just-like-persistent-queue-using-the-in-memory-queue/383453/3 "2025-11-16T13:48:28Z")

</div>

> [@Oscar\_Moyeda](#):
>
> I would like to know if there is a way to have the In-memory Queue batch events like Persistent Queue does? What I want to achieve is to have less small requests, and instead have a single request or batch with all those events (until they fill `pipeline.batch.size: 100` for example).

Is the 100 e/s the real number you will work or is this just for test? Because it may be too small to troubleshoot, it is even smaller than the default batch size of Logstash, which is 125.

Keep in mind that `pipeline.batch.size` is applied per worker, so if you are running logstash on a server with more than one CPU, then it us using more than one worker.

The in-memory queue and the persistent queue works in different ways and have different goals, but in your example by changing to the persisted queue the first change is that you made your pipeline a little slower as everything needs to be written to disk and then read from disk again before being processed.

With a `file` input you are reading events, writing into the page file and then reading it again from the page file before passing through the filters and outputs.

This extra time may have been enough for the events to be grouped on a single batch.

> [@Oscar\_Moyeda](#):
>
> I already tried playing with these pipeline parameters in Logstash.yml:
> 
> - pipeline.batch.size
> - pipeline.batch.delay
> 
> I’ve read that those two are the only parameters that modify the batch behavior, and `pipeline.batch.size` only sets the max number of events per batch, but there might be something I’m missing.

What changes you made to those settings? As you discovered `pipeline.batch.size` only sets the maximum size of the batch, I don't think there is anything else missing.

Try to using these settings:

```auto
pipeline.batch.size: 100
pipeline.batch.delay: 2000

```

And as @Rios mentioned, also set `pipeline.workers: 1` so your pipeline will only use one worker (one CPU core)

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [November 17, 2025, 12:19pm UTC](https://discuss.elastic.co/t/handle-batching-just-like-persistent-queue-using-the-in-memory-queue/383453/4 "2025-11-17T12:19:03Z")

</div>

+Consider also the parameter: [pipeline.ordered](https://www.elastic.co/docs/reference/logstash/logstash-settings-file) , it might be important to you.

---

<div class="post-metadata">

**Author:** ![Oscar\_Moyeda](https://avatars.discourse-cdn.com/v4/letter/o/cdc98d/32.png) [@Oscar\_Moyeda](https://discuss.elastic.co/u/Oscar_Moyeda)\
**Post date:** [November 18, 2025, 6:52pm UTC](https://discuss.elastic.co/t/handle-batching-just-like-persistent-queue-using-the-in-memory-queue/383453/5 "2025-11-18T18:52:43Z")

</div>

This did the trick. Thank you.

---

<div class="post-metadata">

**Author:** ![Oscar\_Moyeda](https://avatars.discourse-cdn.com/v4/letter/o/cdc98d/32.png) [@Oscar\_Moyeda](https://discuss.elastic.co/u/Oscar_Moyeda)\
**Post date:** [November 18, 2025, 6:57pm UTC](https://discuss.elastic.co/t/handle-batching-just-like-persistent-queue-using-the-in-memory-queue/383453/6 "2025-11-18T18:57:50Z")

</div>

First, thanks everyone for your replies, they were very helpful.

> Is the 100 e/s the real number you will work or is this just for test? Because it may be too small to troubleshoot, it is even smaller than the default batch size of Logstash, which is 125.

It was for test indeed.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 18, 2025, 7:14pm UTC](https://discuss.elastic.co/t/handle-batching-just-like-persistent-queue-using-the-in-memory-queue/383453/7 "2025-11-18T19:14:27Z")

</div>

> [@Oscar\_Moyeda](#):
>
> It was for test indeed.

What will be your final output? Elasticsearch?