Elasticsearch output plugin: "workers" setting

Elastic Stack Logstash
#1

Hello,

If I understand well what is described in the [Logstash Processing Pipeline article] (https://www.elastic.co/guide/en/logstash/current/pipeline.html) , it might be that the number of ES output plugin workers (as defined by the 'workers' setting of the ES output plugin) equals to:
number of LS pipeline workers X ES output plugin workers

Could someone confirm please?

In general, how does increasing the "workers" setting of the Elasticsearch output plugin affect the performance? Should it be aligned with the pipeline-workers setting (it is set to ''1' by default)?

Thanks,
MG

#2

Hi,

I noticed that the workers setting in the elasticsearch output plugins directly affects the number of sockets the machine opens towards the elasticsearch nodes.

workers# = socket# per ES host

For example, if you set 3 ES data nodes under the "hosts" variable you will get:
1 socket for each ES host if workers is 1 (default) --> total of 3 open sockets
2 socket for each ES host if workers is 2 --> total of 6 open sockets
3 socket for each ES host if workers is 3 --> total of 9 open sockets

I'm not sure how this relates to the pipeline worker threads. I assume each thread will output the event it is handling to each output plugin it is supposed to reach, in our case the ES output plugin is among the reached ones. Subsequently, I believe the ES output plugin take the event and puts them in a queue, then it will pick from the queue and schedule a free socket to use among the ones it opened (as per the "workers" directive).

I'm not sure of it and would like to deepen my understanding on this specific topic.
Can someone else help us?

Thanks,
Daniele

#3