Hi
Could someone please explain to me how the multicodec works (with pattern ) along with the CSV filter?
In my case I have a problem to handle
26,473,906 lines for one of pattern even though I cranked up the configuration for 32 GB heap memory and 48 workers does not handle my logstash this event.
As I understand that multicode send the bulk of date when it finds the end of the line for a specific pattern.
How do I make the slice to send in parts and free up memory? Is this possible?
You need to share your configuration, the contents of the /usr/share/logstash/pipeline/pipeline_test_db.yml file.
But, if you have a multiline that where the start and end pattern have 26 million lines, I doubt that this will work in Logstash, you need to find other ways to deal with a file this big, probably will need some pre-processing to slice it in smaller files.
Hi @leandrojmp this is the mentioned pipeline
it was crazy but I tried to use even 65 GB heap size
size and ended up failing (memory leak), of course.
warning: thread "[npdb]>worker38" terminated with exception (report_on_exception is true):warning: thread "[npdb]>worker26" terminated with exception (report_on_exception is true):
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.<init>(java/util/ArrayList.java:154)
at org.logstash.common.LsQueueUtils.drain(org/logstash/common/LsQueueUtils.java:78)
at org.logstash.ext.JrubyMemoryReadClientExt.readBatch(org/logstash/ext/JrubyMemoryReadClientExt.java:83)
at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:82)
at jdk.internal.reflect.GeneratedMethodAccessor62.invoke(jdk/internal/reflect/GeneratedMethodAccessor62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)
at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)
at java.lang.invoke.LambdaForm$DMH/0x00007f167d5c9440.invokeVirtual(java/lang/invoke/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x00007f167d5c7440.invoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5c40.reinvoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5440.guard(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5c40.reinvoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f167d5c5440.guard(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300)
at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00007f163ab0a440.invoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.invokeExact_MT(java/lang/invoke/Invokers$Holder)
at org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
at java.lang.Thread.run(java/lang/Thread.java:829)
warning: thread "Ruby-0-Thread-134: :1" terminated with exception (report_on_exception is true):
warning: thread "Ruby-0-Thread-136: :1" terminated with exception (report_on_exception is true):warning: thread "Ruby-0-Thread-147: :1" terminated with exception (report_on_exception is true):
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.