Logstash memory requirements

Elastic Stack Logstash
1

Hi

i am configuring logstash so as to send data from 3 kafka clusters which generates 1lac events per second per each cluster.
can you tell me the java heap memory allocation and pipeline workers and batch size which i should configure.
also let me know how to minimize cpu usage

thanks
soumya

2

You should leave the workers and batch size as the defaults.
Java memory is a different story because it depends on the size of the strings in each event.
workers X events per batch X max event size X 3
The X 3 is because in many cases the strings are duplicated or the event is JSON serialised - so there are multiple copies of various pieces of data in flight.

3

Thanks for the answer guyboertje
Actually i am pushing data from logstash output plugin solr to SOLR cloud collection (here am taking sample cloud collection)
the data was continously passing to solr but suddenly the data passing stopped.
can you tell me why .is it a memory issue.i see no error logs

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.