Multiline codec and csv filter performance

Elastic Stack Logstash
3

Hi @leandrojmp this is the mentioned pipeline
it was crazy but I tried to use even 65 GB heap size
size and ended up failing (memory leak), of course.


warning: thread "[npdb]>worker38" terminated with exception (report_on_exception is true):warning: thread "[npdb]>worker26" terminated with exception (report_on_exception is true):


java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
        at java.util.ArrayList.<init>(java/util/ArrayList.java:154)
        at org.logstash.common.LsQueueUtils.drain(org/logstash/common/LsQueueUtils.java:78)
        at org.logstash.ext.JrubyMemoryReadClientExt.readBatch(org/logstash/ext/JrubyMemoryReadClientExt.java:83)
        at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:82)
        at jdk.internal.reflect.GeneratedMethodAccessor62.invoke(jdk/internal/reflect/GeneratedMethodAccessor62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)
        at java.lang.invoke.LambdaForm$DMH/0x00007f167d5c9440.invokeVirtual(java/lang/invoke/LambdaForm$DMH)
        at java.lang.invoke.LambdaForm$MH/0x00007f167d5c7440.invoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5c40.reinvoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5440.guard(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5c40.reinvoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5440.guard(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300)
        at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder)
        at java.lang.invoke.LambdaForm$MH/0x00007f163ab0a440.invoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.Invokers$Holder.invokeExact_MT(java/lang/invoke/Invokers$Holder)
        at org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
        at java.lang.Thread.run(java/lang/Thread.java:829)
warning: thread "Ruby-0-Thread-134: :1" terminated with exception (report_on_exception is true):
warning: thread "Ruby-0-Thread-136: :1" terminated with exception (report_on_exception is true):warning: thread "Ruby-0-Thread-147: :1" terminated with exception (report_on_exception is true):