Logstash to Elastic Multiple Connections

[yago82]

Hello everyone,

I was wandering if you can help in this particular matter: the actual situation is that we maintain a large Elastic Stack (15+ nodes) with an ingestion workflow composed by two Logstash server on VMs, with multi-instances of logstash per macro-flow. Later in these weeks, we noticed a performance degradation ended in an ingestion delay till 30 minutes on some indices. In the specific, the nodes that owned the Primary Shards of the mentioned indices presented an anormal system load.

Said that, the main idea and linked question is: if we add more Logstash maybe we will be able to split the input load before Elastic, in order to have smaller bunches of data to send to the Cluster per single Logstash? And once we have more Logstash than before, the number of connections could be too much to handle for Elastic or could be a viable solution?

In brief: more connections with smaller data volumes impact more or less than have a few connections but with higher data volume per single Losgstash?

Additional details:

• The Logstash pipelines we are currently using are not performing heavy processing filters on the data.
• The data volumes are huge, we are talking about MB/s of data input on Logstash (but reading from a Kafka).
• The measure of the heap is xmx/xms=1g per single Logstash instance, xmx/xms=30g per single Elastic instance.
• The CPUs are not a problem. Logstash servers have 8 CPU and Elastic nodes have 50+ CPU per node.

Thanks.
Regards.

[stephenb (Stephen Brown)]

Hi @yago82
What version are you on?
and how many primary shards do you have?

If you only have the default 1 primary shard, that could be your limiting factor and could be the reason for the hot spots of load on specific nodes.

Perhaps try 4 or 6 primary shards to increase throughput parallelization, That is often one of the first recommendations.

In your logstash elasticsearch output hosts you can also list all the data nodes.