# Logstash Batch Size/Workers log message **URL:** \ **Category:** Logstash\ **Created:** [March 1, 2016, 11:17pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175 "2016-03-01T23:17:29Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![rahulnadella](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulnadella/32/22117_2.png) [@rahulnadella](https://discuss.elastic.co/u/rahulnadella)\ **Post date:** [March 1, 2016, 11:17pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/1 "2016-03-01T23:17:29Z")
Is there a way to work around the message below I am seeing in my logs? I currently working with Logstash 2.2 and am trying to improve the performance of the number of events running through the pipeline. Currently am using Redis as my input (threads =\> 10 and batch\_count =\> 1000) and Elasticsearch as my output (flush\_size 5000). My current events per second is between 4500/sec - 5000/sec. I am trying to increase the number of events processed by Redis queue. **{:timestamp=\>"2016-03-01T16:01:25.001000-0500", :message=\>"CAUTION: Recommended inflight events max exceeded! Logstash will run with up to 15000 events in memory in your current configuration. If your message sizes are large this may cause instability with the default heap size. Please consider setting a non-standard heap size, changing the batch size (currently 1000), or changing the number of pipeline workers (currently 15)", :level=\>:warn}** ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 2, 2016, 1:36am UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/2 "2016-03-02T01:36:52Z")
I'd reduce the ES batch size, larger is not always better in this instance. ---
**Author:** ![clement](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement/32/6895_2.png) [@clement](https://discuss.elastic.co/u/clement)\ **Post date:** [April 8, 2016, 1:33pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/3 "2016-04-08T13:33:51Z")
Hello warkolm Where can i modify the ES batch size ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 9, 2016, 12:39am UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/4 "2016-04-09T00:39:45Z")
[https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-flush\_size](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-flush_size) ---
**Author:** ![clement](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement/32/6895_2.png) [@clement](https://discuss.elastic.co/u/clement)\ **Post date:** [October 6, 2016, 1:46pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/5 "2016-10-06T13:46:00Z")
Hi Warkolm i read the doc but i did not find wher modify th ES batch size could you help me ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 6, 2016, 1:58pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/6 "2016-10-06T13:58:57Z")
How many [workers](https://www.elastic.co/guide/en/logstash/2.2/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-workers) do you have configured for the Elasticsearch output? How many workers is Logstash running with? ---
**Author:** ![clement](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement/32/6895_2.png) [@clement](https://discuss.elastic.co/u/clement)\ **Post date:** [October 6, 2016, 2:48pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/7 "2016-10-06T14:48:28Z")
2048 for each. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 6, 2016, 3:54pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/8 "2016-10-06T15:54:18Z")
2048?? That seems extremely excessive and inefficient. I believe good starting point is to set it to the number of cores on the host, and then potentially slowly increase until you see no further improvement in throughput. That should land you at a considerably smaller number... I would also recommend starting with a smaller batch and flush size, e.g. 1000, and then gradually increase this as long as it improves throughput. ---
**Author:** ![clement](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement/32/6895_2.png) [@clement](https://discuss.elastic.co/u/clement)\ **Post date:** [October 6, 2016, 4:08pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/9 "2016-10-06T16:08:36Z")
Hello More i improve the workers on logstash and in output elasticsearch more i send data. here my filebeat log 🙂 2016-10-06T18:02:43+02:00 INFO Events sent: 2048 2016-10-06T18:02:43+02:00 INFO Registry file updated. 2 states written. 2016-10-06T18:04:05+02:00 INFO Events sent: 2048 2016-10-06T18:04:05+02:00 INFO Registry file updated. 2 states written. 2016-10-06T18:06:09+02:00 INFO Events sent: 2048 2016-10-06T18:06:09+02:00 INFO Registry file updated. 2 states written. 2016-10-06T18:07:02+02:00 INFO Events sent: 2048 2016-10-06T18:07:02+02:00 INFO Registry file updated. 2 states written. 2016-10-06T18:08:13+02:00 INFO Events sent: 2048 2016-10-06T18:08:13+02:00 INFO Registry file updated. 2 states written. Here my data flux on logstash server. ![](https://us1.discourse-cdn.com/elastic/original/2X/9/9cc3bb95d52c5b1d16034250e4cf60f44de9dcc7.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 6, 2016, 4:58pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/10 "2016-10-06T16:58:39Z")
Are you talking about batch size or actual input and filter worker threads? Can you share the Logstash config? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 7, 2016, 1:33am UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/11 "2016-10-07T01:33:52Z")
Actually @clement, please make a new thread 🙂 ---
**Author:** ![clement](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement/32/6895_2.png) [@clement](https://discuss.elastic.co/u/clement)\ **Post date:** [October 7, 2016, 7:29am UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/12 "2016-10-07T07:29:16Z")
/etc/sysconfig/logstash LS\_HOME=/var/lib/logstash LS\_OPTS="-w 8" LS\_OPEN\_FILES=163840 LS\_NICE=19 KILL\_ON\_STOP\_TIMEOUT=0 /etc/logstash/conf.d/output\_elasticsearch.conf output { elasticsearch { hosts =\> ["node1:9200","node2:9200","node3:9200"] index =\> "logstash-%{type}-%{+YYYY.MM.dd}" flush\_size =\> 500000 } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 7, 2016, 7:38am UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/13 "2016-10-07T07:38:54Z")
> [@clement](#): > > LS\_OPTS="-w 8" OK, so you are using 8 worker threads, not 20148. That is much kore reasonable. The flush size is however excessive and most likely inefficient. I would recommend lowering it to 1000 or 5000 and setting the [workers](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-workers) parameter in the Elasticsearch output plugin to 8 as well in order to use more connections to Elasticsearch in parallel. At the moment you are just sending very large bulk requests across a single connection, which is not very efficient. Start with that as a baseline and then gradually tune the batch size until you see no further improvement in throughput. ---
**Author:** ![clement](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement/32/6895_2.png) [@clement](https://discuss.elastic.co/u/clement)\ **Post date:** [October 7, 2016, 12:25pm UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/14 "2016-10-07T12:25:56Z")
Hi I modified my conf like this flush\_size =\> 5000 workers =\> 8 but my transfer steel slow for applicatives log 🙂 2016-10-07T13:54:17+02:00 INFO Registry file updated. 2 states written. 2016-10-07T13:56:26+02:00 INFO Events sent: 2048 2016-10-07T13:56:26+02:00 INFO Registry file updated. 2 states written. 2016-10-07T13:58:43+02:00 INFO Events sent: 2048 2016-10-07T13:58:43+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:01:03+02:00 INFO Events sent: 2048 2016-10-07T14:01:03+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:03:16+02:00 INFO Events sent: 2048 2016-10-07T14:03:16+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:05:41+02:00 INFO Events sent: 2048 2016-10-07T14:05:41+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:07:43+02:00 INFO Events sent: 2048 2016-10-07T14:07:43+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:09:34+02:00 INFO Events sent: 2048 2016-10-07T14:09:34+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:11:29+02:00 INFO Events sent: 2048 2016-10-07T14:11:29+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:13:40+02:00 INFO Events sent: 2048 2016-10-07T14:13:40+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:15:43+02:00 INFO Events sent: 2048 2016-10-07T14:15:43+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:17:49+02:00 INFO Events sent: 2048 2016-10-07T14:17:49+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:19:48+02:00 INFO Events sent: 2048 2016-10-07T14:19:48+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:21:33+02:00 INFO Events sent: 2048 2016-10-07T14:21:33+02:00 INFO Registry file updated. 2 states written. 2016-10-07T14:23:59+02:00 INFO Events sent: 2048 2016-10-07T14:23:59+02:00 INFO Registry file updated. 2 states written. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:35am UTC](https://discuss.elastic.co/t/logstash-batch-size-workers-log-message/43175/15 "2017-07-06T04:35:14Z")