Can ruby code be a speed bottleneck in logstash config file?

Elastic Stack Logstash
1

I m trying to get as much throughput as possible from my setup. I am using the http input. The client sometimes sends a collection of events instead of a single event for efficiency sake. Here is the relevant thread with more information. I use the same code to separate the events.

filter
{
        split
        {
                field => "events"
        }

        ruby
        {
                code => "
                event.get('events').each do |k, v|
                event.set(k, v)
                end
                "
        }
}

Now this ruby code will be invoked for every event I process. Can it be a speed bottleneck? Is is possible to do something similar using existing logstash filters so that I can compare?

2

logstash filters are generally written in ruby (some also call Java functions). It is unlikely that code in a ruby filter will underperform ruby code in any other filter.

1 Like
3

Thanks for the prompt reply. The remaining option then is to increase the number of pipelines. I have persistence enabled. I recall that it means that pipeline have a single worker thread as a result. I will clone and increase the number of pipelines. Is there any limit on number of pipelines I can have?

My machine (hosts only logstash) is a 500GB disk and 16GB RAM machine with 8GB allocated to Logstash.

4

I do not think persistence requires pipeline.workers 1.

5

I was going through this blog.
"If the persistent queue is enabled for a pipeline, then Logstash will run a single threaded persistent queue for that pipeline — the persistent queue does not run across multiple threads within a single pipeline. "

Maybe I mis-interpreted it. I will try to increase the number of pipeline.workers in pipelines.yml and see if the throughput increases.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.