Make batch size configurable in elastic search output plugin

Elastic Stack Logstash
1

Is there a way to configure the batch size in the elastic search output plugin?

I have an Elastic Search server at AWS on nodes with a request max size set to 10MB. If the output plugin sends more than 10MB in a request, then I would lose data. Based on the documentation the batches are capped at 20MB which would cause problems for me.

According to this thread at AWS there is currently no way to configure this in AWS.

2

Which version of Logstash are you using?

3

I'm using version 6.2.2 of Logstash.
Elastic Search in AWS is also running 6.2.2

Would the logstash setting pipeline.batch.size help here at all?

4

I believe the internal batch size used will be the maximum number of documents that will be sent to Elasticsearch as part off a single bulk request.

5

So the Elastic Search output plugin will not combine several processing batches when sending bulk inserts to Elastic Search? In other words, pipeline.batch.size would save the day. Correct?

6

Yes, but that is measured in number of documents, not total size of the bulk request.

7

Nice, i think that solves my problem. It's at least something that i can fiddle with and lower when needed.

Just to make sure we're on the same page. A document here is a JSON object. You're not referring some other kind of document?

8

I am referring to an event in Logstash, which will get turned into a JSON document. One thing that I guess could increase the size of the bulk request would be if you created new events as part of the batch, e.g. using ruby or the clone filter.

9

Ah, very good point! Thanks a lot Christian!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.