# Multiline codec and csv filter performance **URL:** \ **Category:** Logstash\ **Created:** [July 22, 2022, 10:16am UTC](https://discuss.elastic.co/t/multiline-codec-and-csv-filter-performance/310383 "2022-07-22T10:16:40Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [July 22, 2022, 10:16am UTC](https://discuss.elastic.co/t/multiline-codec-and-csv-filter-performance/310383/1 "2022-07-22T10:16:40Z")
Hi Could someone please explain to me how the multicodec works (with pattern ) along with the CSV filter? In my case I have a problem to handle 26,473,906 lines for one of pattern even though I cranked up the configuration for 32 GB heap memory and 48 workers does not handle my logstash this event. currently config: ```auto - pipeline.id: test_db path.config: "/usr/share/logstash/pipeline/pipeline_test_db.yml" pipeline.workers: 48 pipeline.batch.size: 2000 pipeline.batch.delay: 50 pipeline.ordered: auto ``` As I understand that multicode send the bulk of date when it finds the end of the line for a specific pattern. How do I make the slice to send in parts and free up memory? Is this possible? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 22, 2022, 2:09pm UTC](https://discuss.elastic.co/t/multiline-codec-and-csv-filter-performance/310383/2 "2022-07-22T14:09:18Z")
You need to share your configuration, the contents of the `/usr/share/logstash/pipeline/pipeline_test_db.yml` file. But, if you have a multiline that where the start and end pattern have 26 million lines, I doubt that this will work in Logstash, you need to find other ways to deal with a file this big, probably will need some pre-processing to slice it in smaller files. ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [July 22, 2022, 7:51pm UTC](https://discuss.elastic.co/t/multiline-codec-and-csv-filter-performance/310383/3 "2022-07-22T19:51:54Z")
Hi @leandrojmp this is the mentioned [pipeline](https://discuss.elastic.co/t/csv-plugin-cooperating-with-multiplying-pattern/309865/10) it was crazy but I tried to use even 65 GB heap size size and ended up failing (memory leak), of course. ```auto warning: thread "[npdb]>worker38" terminated with exception (report_on_exception is true):warning: thread "[npdb]>worker26" terminated with exception (report_on_exception is true): java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space at java.util.ArrayList.(java/util/ArrayList.java:154) at org.logstash.common.LsQueueUtils.drain(org/logstash/common/LsQueueUtils.java:78) at org.logstash.ext.JrubyMemoryReadClientExt.readBatch(org/logstash/ext/JrubyMemoryReadClientExt.java:83) at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:82) at jdk.internal.reflect.GeneratedMethodAccessor62.invoke(jdk/internal/reflect/GeneratedMethodAccessor62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566) at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441) at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305) at java.lang.invoke.LambdaForm$DMH/0x00007f167d5c9440.invokeVirtual(java/lang/invoke/LambdaForm$DMH) at java.lang.invoke.LambdaForm$MH/0x00007f167d5c7440.invoke(java/lang/invoke/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5c40.reinvoke(java/lang/invoke/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5440.guard(java/lang/invoke/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5c40.reinvoke(java/lang/invoke/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5440.guard(java/lang/invoke/LambdaForm$MH) at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder) at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300) at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder) at java.lang.invoke.LambdaForm$MH/0x00007f163ab0a440.invoke(java/lang/invoke/LambdaForm$MH) at java.lang.invoke.Invokers$Holder.invokeExact_MT(java/lang/invoke/Invokers$Holder) at org.jruby.RubyProc.call(org/jruby/RubyProc.java:318) at java.lang.Thread.run(java/lang/Thread.java:829) warning: thread "Ruby-0-Thread-134: :1" terminated with exception (report_on_exception is true): warning: thread "Ruby-0-Thread-136: :1" terminated with exception (report_on_exception is true):warning: thread "Ruby-0-Thread-147: :1" terminated with exception (report_on_exception is true): ``` ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [July 22, 2022, 7:53pm UTC](https://discuss.elastic.co/t/multiline-codec-and-csv-filter-performance/310383/4 "2022-07-22T19:53:49Z")
the processing file consists with such so You can see that such file has a lot of rows with DNs ```auto ||Line 1: # snapshot,66760085,20220721044503| |---|---| ||Line 2: # Network Entities| ||Line 3440: # Imsis| ||Line 3441: # DNs| ||Line 26477347: # DN Blocks| ||Line 26479335: # Numbers| ||Line 26798461: # Number Blocks| ||Line 26798462: # 20220721051007| ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 19, 2022, 7:53pm UTC](https://discuss.elastic.co/t/multiline-codec-and-csv-filter-performance/310383/5 "2022-08-19T19:53:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.