Can ruby code be a speed bottleneck in logstash config file?

I m trying to get as much throughput as possible from my setup. I am using the http input. The client sometimes sends a collection of events instead of a single event for efficiency sake. Here is the relevant thread with more information. I use the same code to separate the events.

                field => "events"

                code => "
                event.get('events').each do |k, v|
                event.set(k, v)

Now this ruby code will be invoked for every event I process. Can it be a speed bottleneck? Is is possible to do something similar using existing logstash filters so that I can compare?

logstash filters are generally written in ruby (some also call Java functions). It is unlikely that code in a ruby filter will underperform ruby code in any other filter.

Thanks for the prompt reply. The remaining option then is to increase the number of pipelines. I have persistence enabled. I recall that it means that pipeline have a single worker thread as a result. I will clone and increase the number of pipelines. Is there any limit on number of pipelines I can have?

My machine (hosts only logstash) is a 500GB disk and 16GB RAM machine with 8GB allocated to Logstash.

I do not think persistence requires pipeline.workers 1.

I was going through this blog.
"If the persistent queue is enabled for a pipeline, then Logstash will run a single threaded persistent queue for that pipeline — the persistent queue does not run across multiple threads within a single pipeline. "

Maybe I mis-interpreted it. I will try to increase the number of pipeline.workers in pipelines.yml and see if the throughput increases.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.