How to tune Logstash to Elasticsearch shipping

@danygor:

For the current versions of the LS in the ES output workers is the number of output instances in use.
A single ES output is not yet threadsafe so it cannot be shared amongst the worker threads.
Currently we create a "pool" of outputs accessed via a blocking queue - so if ES workers = 1 then there is a pool of one output and only one worker can be communicating with ES at any given time, the other threads must wait.
This is why you don't see much improvement in throughput when increasing the batch count or pipeline workers without a corresponding increase in the ES worker count.

Thanks for completing the tests. We see a plateau of 20k eps at about 16 to 18 workers and outputs.

As far as I know, in ES, the bulk_request http call only returns when a quorum of of active shards indexes the documents.

So 20k eps at 3000 docs per request translates to 6.66667 ES requests per second or 150 ms round trip time.

Now its time to tune ES.

See:
bulk-consistency
performance-considerations