How many logstash can connect to elasticsearch cluster?

Elastic Stack Elasticsearch
1

we have logstash with 4 pipeline and each pipeline.worker is set to 120 and Elasticsearch cluster has 6 data nodes.(c5.9xlarge)

we are facing high cpu usage in Elasticsearch data nodes when we have more than 10 logstash.

what is the max workers or logstash can we have?

2

Why would you have 120 workers configured per pipeline? That sounds excessive and I would recommend reducing that significantly so the total number of workers roughly matches the number of CPU cores on the host where it runs. Larger number does not necessarily equal better performance.

If you want to improve performance instead look at increasing the batch size a bit but do not go crazy.

There is no hard limit with respect to how many connections a node can handle as it depends on the load and how many indices and shards each bulk requests targets. You will need to test.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.