I'm trying to get the hang off tweaking logstashs performance. I searched the net but unfortunately, I can't seem to find a good guide on how to tweak and actually measure the performance of a pipeline.
I can't use the pipeline management stuff inside Kibana which is why I'm doing with a curl pipelines command.
Currently, my problem is the following:
I got a dedicated redis and a dedicated logstash cluster. The logstash cluster pulls the data from the redis cluster, does it's thing and then writes to a dedicated elasticsearch cluster.
Even if the redis cluster has many keys "building up" or waiting to get pulled, the CPU utilization of the logstash cluster is only at ~ 50%.
One thing I found is by diving the queue_push_duration_in_millis through duration_in_millis, I should be able to see if Logstash is waiting for workers. I sometimes get a value of 4,X which would mean logstash is waiting a lot of its time, correct?
I'm unsure how I could tweak the settings and would really love a useful guide which I could use.
Thanks in advance