# What kind of size is better to set logstash pipline?

**URL:** <https://discuss.elastic.co/t/what-kind-of-size-is-better-to-set-logstash-pipline/95112>\
**Category:** Logstash\
**Created:** [July 31, 2017, 4:25am UTC](https://discuss.elastic.co/t/what-kind-of-size-is-better-to-set-logstash-pipline/95112 "2017-07-31T04:25:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Robin\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_guo/32/42297_2.png) [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\
**Post date:** [July 31, 2017, 4:25am UTC](https://discuss.elastic.co/t/what-kind-of-size-is-better-to-set-logstash-pipline/95112/1 "2017-07-31T04:25:26Z")

</div>

I have 3 logstash servers with **16GB memory** and **4vcpu** enabled in a cluster environment.  
so my question is that what kind of size is better to set logstash pipline ?

My configure file like this as below:

**OS:**

```
mem:16G
cpu:4

```

**Java heap size:**

```
-Xms8g
-Xmx8g

```

**logstash pipline:**

```
# This defaults to the number of the host's CPU cores.
pipeline.workers: 4
 
# How many workers should be used per output plugin instance
pipeline.output.workers: 4
 
# How many events to retrieve from inputs before sending to filters+workers
pipeline.batch.size: 5000

```

**Problem:**

`[2017-07-31T11:59:45,748][WARN][logstash.pipeline] CAUTION: Recommended inflight events max exceeded! Logstash will run with up to 20000 events in memory in your current configuration. If your message sizes are large this may cause instability with the default heap size. Please consider setting a non-standard heap size, changing the batch size (currently 5000), or changing the number of pipeline workers (currently 4)`

Ａny ideas of what kind of size is better to set logstash pipline ?  
by the way, what's max pipline size of single instance logstash even if with a big memory or multiple core cpus?

Thanks in adavance

---

<div class="post-metadata">

**Author:** ![Robin\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_guo/32/42297_2.png) [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\
**Post date:** [August 1, 2017, 2:08am UTC](https://discuss.elastic.co/t/what-kind-of-size-is-better-to-set-logstash-pipline/95112/2 "2017-08-01T02:08:16Z")

</div>

Any update on this?

---

<div class="post-metadata">

**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\
**Post date:** [August 1, 2017, 12:20pm UTC](https://discuss.elastic.co/t/what-kind-of-size-is-better-to-set-logstash-pipline/95112/3 "2017-08-01T12:20:03Z")

</div>

First of all, this warning is just this. A warning that the total inflight events would be more than 10k (although depending on the message size it may run out of heap). Logstash should start fine with 5k batch sizes even it seems overkill to me.

Why did you opt for such a large batch size? Bigger does not necessarily equal better. The answer to your question is "test performance".  
You should monitor your CPU utilization and I/O wait as well as Logstash throughput, first starting with the default batch size, and then slowly increment it until you hit a maximum throughtput while minimizing I/O wait. There is not magic formula that works across all use cases.

---

<div class="post-metadata">

**Author:** ![Robin\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_guo/32/42297_2.png) [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\
**Post date:** [August 4, 2017, 2:28am UTC](https://discuss.elastic.co/t/what-kind-of-size-is-better-to-set-logstash-pipline/95112/4 "2017-08-04T02:28:26Z")

</div>

hi @paz  
Thanks for you suggestions, i'll enable jvm monitoring on logstash server.  
i'll also jvm performance besides cpu and io etc..

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 1, 2017, 2:28am UTC](https://discuss.elastic.co/t/what-kind-of-size-is-better-to-set-logstash-pipline/95112/5 "2017-09-01T02:28:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.