# Make batch size configurable in elastic search output plugin **URL:** \ **Category:** Logstash\ **Created:** [March 27, 2018, 9:42am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724 "2018-03-27T09:42:24Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Patrik\_Iselind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrik_iselind/32/25744_2.png) [@Patrik\_Iselind](https://discuss.elastic.co/u/Patrik_Iselind)\ **Post date:** [March 27, 2018, 9:42am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/1 "2018-03-27T09:42:24Z")
Is there a way to configure the [batch size](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_batch_sizes) in the elastic search output plugin? I have an Elastic Search server at AWS on nodes with a request max size set to 10MB. If the output plugin sends more than 10MB in a request, then I would lose data. Based on the documentation the batches are capped at 20MB which would cause problems for me. According to [this](https://forums.aws.amazon.com/thread.jspa?messageID=802516) thread at AWS there is currently no way to configure this in AWS. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 27, 2018, 9:46am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/2 "2018-03-27T09:46:57Z")
Which version of Logstash are you using? ---
**Author:** ![Patrik\_Iselind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrik_iselind/32/25744_2.png) [@Patrik\_Iselind](https://discuss.elastic.co/u/Patrik_Iselind)\ **Post date:** [March 27, 2018, 9:47am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/3 "2018-03-27T09:47:48Z")
I'm using version 6.2.2 of Logstash. Elastic Search in AWS is also running 6.2.2 Would the logstash setting `pipeline.batch.size` help here at all? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 27, 2018, 9:49am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/4 "2018-03-27T09:49:42Z")
I believe the internal batch size used will be the maximum number of documents that will be sent to Elasticsearch as part off a single bulk request. ---
**Author:** ![Patrik\_Iselind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrik_iselind/32/25744_2.png) [@Patrik\_Iselind](https://discuss.elastic.co/u/Patrik_Iselind)\ **Post date:** [March 27, 2018, 9:51am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/5 "2018-03-27T09:51:39Z")
So the Elastic Search output plugin will not combine several processing batches when sending bulk inserts to Elastic Search? In other words, `pipeline.batch.size` would save the day. Correct? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 27, 2018, 9:53am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/6 "2018-03-27T09:53:27Z")
Yes, but that is measured in number of documents, not total size of the bulk request. ---
**Author:** ![Patrik\_Iselind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrik_iselind/32/25744_2.png) [@Patrik\_Iselind](https://discuss.elastic.co/u/Patrik_Iselind)\ **Post date:** [March 27, 2018, 9:57am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/7 "2018-03-27T09:57:34Z")
Nice, i think that solves my problem. It's at least something that i can fiddle with and lower when needed. Just to make sure we're on the same page. A document here is a JSON object. You're not referring some other kind of document? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 27, 2018, 9:59am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/8 "2018-03-27T09:59:51Z")
I am referring to an event in Logstash, which will get turned into a JSON document. One thing that I guess could increase the size of the bulk request would be if you created new events as part of the batch, e.g. using ruby or the clone filter. ---
**Author:** ![Patrik\_Iselind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrik_iselind/32/25744_2.png) [@Patrik\_Iselind](https://discuss.elastic.co/u/Patrik_Iselind)\ **Post date:** [March 27, 2018, 10:00am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/9 "2018-03-27T10:00:42Z")
Ah, very good point! Thanks a lot Christian! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 24, 2018, 10:00am UTC](https://discuss.elastic.co/t/make-batch-size-configurable-in-elastic-search-output-plugin/125724/10 "2018-04-24T10:00:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.