# Can ruby code be a speed bottleneck in logstash config file?

**URL:** <https://discuss.elastic.co/t/can-ruby-code-be-a-speed-bottleneck-in-logstash-config-file/270395>\
**Category:** Logstash\
**Created:** [April 16, 2021, 1:28pm UTC](https://discuss.elastic.co/t/can-ruby-code-be-a-speed-bottleneck-in-logstash-config-file/270395 "2021-04-16T13:28:27Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\
**Post date:** [April 16, 2021, 1:28pm UTC](https://discuss.elastic.co/t/can-ruby-code-be-a-speed-bottleneck-in-logstash-config-file/270395/1 "2021-04-16T13:28:27Z")

</div>

I m trying to get as much throughput as possible from my setup. I am using the http input. The client sometimes sends a collection of events instead of a single event for efficiency sake. Here is the [relevant thread with more information](https://stackoverflow.com/questions/41746502/serilog-http-sink-logstash-splitting-serilog-message-array-into-individual-lo). I use the same code to separate the events.

```
filter
{
        split
        {
                field => "events"
        }

        ruby
        {
                code => "
                event.get('events').each do |k, v|
                event.set(k, v)
                end
                "
        }
}

```

Now this ruby code will be invoked for every event I process. Can it be a speed bottleneck? Is is possible to do something similar using existing logstash filters so that I can compare?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 16, 2021, 2:31pm UTC](https://discuss.elastic.co/t/can-ruby-code-be-a-speed-bottleneck-in-logstash-config-file/270395/2 "2021-04-16T14:31:15Z")

</div>

> [@pk.241011](#):
>
> Is is possible to do something similar using existing logstash filters

logstash filters are generally written in ruby (some also call Java functions). It is unlikely that code in a ruby filter will underperform ruby code in any other filter.

---

<div class="post-metadata">

**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\
**Post date:** [April 16, 2021, 10:34pm UTC](https://discuss.elastic.co/t/can-ruby-code-be-a-speed-bottleneck-in-logstash-config-file/270395/3 "2021-04-16T22:34:49Z")

</div>

Thanks for the prompt reply. The remaining option then is to increase the number of pipelines. I have persistence enabled. I recall that it means that pipeline have a single worker thread as a result. I will clone and increase the number of pipelines. Is there any limit on number of pipelines I can have?

My machine (hosts only logstash) is a 500GB disk and 16GB RAM machine with 8GB allocated to Logstash.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 16, 2021, 11:39pm UTC](https://discuss.elastic.co/t/can-ruby-code-be-a-speed-bottleneck-in-logstash-config-file/270395/4 "2021-04-16T23:39:49Z")

</div>

> [@pk.241011](#):
>
> I recall that it means that pipeline have a single worker thread as a result.

I do not think persistence requires pipeline.workers 1.

---

<div class="post-metadata">

**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\
**Post date:** [April 17, 2021, 3:21am UTC](https://discuss.elastic.co/t/can-ruby-code-be-a-speed-bottleneck-in-logstash-config-file/270395/5 "2021-04-17T03:21:09Z")

</div>

I was going through this [blog](https://www.elastic.co/blog/using-parallel-logstash-pipelines-to-improve-persistent-queue-performance).  
"If the persistent queue is enabled for a pipeline, then Logstash will run a single threaded persistent queue for that pipeline — the persistent queue does not run across multiple threads within a single pipeline. "

Maybe I mis-interpreted it. I will try to increase the number of pipeline.workers in pipelines.yml and see if the throughput increases.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 15, 2021, 3:21am UTC](https://discuss.elastic.co/t/can-ruby-code-be-a-speed-bottleneck-in-logstash-config-file/270395/6 "2021-05-15T03:21:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.