Logstash elasticsearch pipeline processing rate

Elastic Stack Logstash
1

i have a basic question on logstash elasticsearch pipeline processing rate.
When i set flush_size to a value, does logstash wait until that size is reached before it starts sending it to es? if the size is not reached what happens?
how do i make it work for bulk and also non bulk requests dynamically based on size of the input.

Thanks

2

The documentation of the idle_flush_time option should answer your questions.

3

does increasing number of logstash consumers increase the indexing speed?

4

That depends on what is limiting performance. If any of the outputs, e.g. Elasticsearch, is not able to handle higher throughput, adding Logstash instances may not help. If however Logstash, which usually is CPU bound, is the bottleneck, scaling up/out the Logstash layer may help.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.