Dear community,
I have a log file that contains multiple entries per transaction id
Entries are ordered in file, but not necessary one after each other
I tried using logstash aggregate filter as follow : My idea is to get the status of the lastest entry and replicate it to other entries for each transaction id.
aggregate {
task_id => "%{transaction_id}"
code => "
map['table'] ||= [];
map['table'].push(event.get('transaction_attributes_hash'));
map['last_status'] = event.get('transaction_status');
"
push_map_as_event_on_timeout => true
timeout => 60
inactivity_timeout => 5
}
#Split as all we need is to add last log status to previous logs
if([table]){
split { field => "[table]"}
}
else{
#drop the previous event
drop{}
}
My pipelines.yml is as follow
- pipeline.id: pipeline
queue.type: persisted
path.config: "/code/***/***/conf.d/*.conf"
pipeline.workers: 1
pipeline.ordered: true
pipeline.batch.size: 25
My jvm.options is as follow
-Xms128m
-Xmx128m
When i run the pipeline for 2 input json files (earch one of size 15MB & 10000 logs), i got java.lang.OutOfMemoryError: Java heap space
I changed jvm.options is as follow; but got same error
-Xms256m
-Xmx256m
Can someone please advice what's the origin of this error ?
THank You
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to E:/elastic_stack/logstash-7.15.0/logs which is now configured via log4j2.properties
[2022-03-08T12:09:50,330][INFO ][logstash.runner ] Log4j configuration path used is: E:\elastic_stack\logstash-7.15.0\config\log4j2.properties
[2022-03-08T12:09:50,341][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [mswin32-x86_64]"}
[2022-03-08T12:09:52,243][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2022-03-08T12:09:54,485][INFO ][org.reflections.Reflections] Reflections took 85 ms to scan 1 urls, producing 120 keys and 417 values
[2022-03-08T12:09:56,408][INFO ][logstash.filters.ruby.script] Test run complete {:script_path=>"/code/***/***/ruby/code_ruby.rb", :results=>{:passed=>0, :failed=>0, :errored=>0}}
[2022-03-08T12:09:56,766][INFO ][logstash.outputs.elasticsearch][****] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//****:9200"]}
[2022-03-08T12:09:57,162][INFO ][logstash.outputs.elasticsearch][****] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@****:9200/]}}
[2022-03-08T12:09:57,646][WARN ][logstash.outputs.elasticsearch][****] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@****:9200/"}
[2022-03-08T12:09:57,739][INFO ][logstash.outputs.elasticsearch][****] Elasticsearch version determined (7.13.2) {:es_version=>7}
[2022-03-08T12:09:57,743][WARN ][logstash.outputs.elasticsearch][****] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-03-08T12:09:57,888][INFO ][logstash.outputs.elasticsearch][****] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//****:9200"]}
[2022-03-08T12:09:57,927][INFO ][logstash.outputs.elasticsearch][****] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@****:9200/]}}
[2022-03-08T12:09:58,003][INFO ][logstash.outputs.elasticsearch][****] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-03-08T12:09:58,104][WARN ][logstash.outputs.elasticsearch][****] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@****:9200/"}
[2022-03-08T12:09:58,160][INFO ][logstash.outputs.elasticsearch][****] Elasticsearch version determined (7.13.2) {:es_version=>7}
[2022-03-08T12:09:58,161][WARN ][logstash.outputs.elasticsearch][****] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-03-08T12:09:58,302][INFO ][logstash.outputs.elasticsearch][****] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-03-08T12:09:58,588][INFO ][logstash.javapipeline ][****] Starting pipeline {:pipeline_id=>"****", "pipeline.workers"=>1, "pipeline.batch.size"=>1000, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["E:/code/***/***/conf.d/10-****-input.conf", "E:/code/***/***/conf.d/20-****-filters-01.conf", "E:/code/***/***/conf.d/20-****-filters-02.conf", "E:/code/***/***/conf.d/30-****-output.conf"], :thread=>"#<Thread:0x55667ce7 run>"}
[2022-03-08T12:10:00,020][INFO ][logstash.javapipeline ][****] Pipeline Java execution initialization time {"seconds"=>1.43}
[2022-03-08T12:10:00,120][INFO ][logstash.javapipeline ][****] Pipeline started {"pipeline.id"=>"****"}
[2022-03-08T12:10:00,169][INFO ][filewatch.observingread ][****][b0dfdaca18ef62b20a27a3bccec19852055439302c5e5b717b2775ab810f02e6] START, creating Discoverer, Watch with file and sincedb collections
[2022-03-08T12:10:00,208][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:"****"], :non_running_pipelines=>[]}
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid4864.hprof ...
Heap dump file created [390261931 bytes in 2.390 secs]
warning: thread "[****]>worker0" terminated with exception (report_on_exception is true):
java.lang.OutOfMemoryError: Java heap space
at org.jruby.RubyString.cat19(org/jruby/RubyString.java:1375)
at org.jruby.RubyString.cat19(org/jruby/RubyString.java:1365)
at org.jruby.ext.stringio.StringIO.stringIOWrite(org/jruby/ext/stringio/StringIO.java:1212)
at org.jruby.ext.stringio.StringIO.write(org/jruby/ext/stringio/StringIO.java:1169)
at java.lang.invoke.LambdaForm$DMH/0x00000001007f2840.invokeVirtual(java/lang/invoke/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x000000010080b440.invoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00000001007cc040.guard(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00000001007cc040.guard(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
at E_3a_.elastic_stack.logstash_minus_7_dot_15_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_0_dot_2_minus_java.lib.logstash.outputs.elasticsearch.http_client.bulk(E:/elastic_stack/logstash-7.15.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:137)
at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x0000000100856c40.invoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.invokeExact_MT(java/lang/invoke/Invokers$Holder)
at org.jruby.RubyEnumerable$EachWithIndex.call(org/jruby/RubyEnumerable.java:1202)
at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1820)
at org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)
at org.jruby.RubyClass.finvokeWithRefinements(org/jruby/RubyClass.java:514)
warning: thread "Ruby-0-Thread-17: :1" terminated with exception (report_on_exception is true):
java.lang.OutOfMemoryError: Java heap space
at org.jruby.RubyString.cat19(org/jruby/RubyString.java:1375)
at org.jruby.RubyString.cat19(org/jruby/RubyString.java:1365)
at org.jruby.ext.stringio.StringIO.stringIOWrite(org/jruby/ext/stringio/StringIO.java:1212)
at org.jruby.ext.stringio.StringIO.write(org/jruby/ext/stringio/StringIO.java:1169)
at java.lang.invoke.LambdaForm$DMH/0x00000001007f2840.invokeVirtual(java/lang/invoke/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x000000010080b440.invoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00000001007cc040.guard(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00000001007cc040.guard(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
at E_3a_.elastic_stack.logstash_minus_7_dot_15_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_0_dot_2_minus_java.lib.logstash.outputs.elasticsearch.http_client.bulk(E:/elastic_stack/logstash-7.15.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:137)
at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x0000000100856c40.invoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.invokeExact_MT(java/lang/invoke/Invokers$Holder)
at org.jruby.RubyEnumerable$EachWithIndex.call(org/jruby/RubyEnumerable.java:1202)
at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1820)
at org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)
at org.jruby.RubyClass.finvokeWithRefinements(org/jruby/RubyClass.java:514)
[2022-03-08T12:10:49,020][INFO ][filewatch.observingread ][****] QUIT - closing all files and shutting down.
[2022-03-08T12:10:49,039][WARN ][logstash.javapipeline ][****] Waiting for input plugin to close {:pipeline_id=>"****", :thread=>"#<Thread:0x55667ce7 run>"}
[2022-03-08T12:10:49,096][FATAL][org.logstash.Logstash ]
java.lang.OutOfMemoryError: Java heap space
at org.jruby.RubyString.cat19(org/jruby/RubyString.java:1375) ~[jruby-complete-9.2.19.0.jar:?]
at org.jruby.RubyString.cat19(org/jruby/RubyString.java:1365) ~[jruby-complete-9.2.19.0.jar:?]
at org.jruby.ext.stringio.StringIO.stringIOWrite(org/jruby/ext/stringio/StringIO.java:1212) ~[jruby-complete-9.2.19.0.jar:?]
at org.jruby.ext.stringio.StringIO.write(org/jruby/ext/stringio/StringIO.java:1169) ~[jruby-complete-9.2.19.0.jar:?]
at java.lang.invoke.LambdaForm$DMH/0x00000001007f2840.invokeVirtual(java/lang/invoke/LambdaForm$DMH) ~[?:?]
at java.lang.invoke.LambdaForm$MH/0x000000010080b440.invoke(java/lang/invoke/LambdaForm$MH) ~[?:?]
at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder) ~[?:?]
at java.lang.invoke.LambdaForm$MH/0x00000001007cc040.guard(java/lang/invoke/LambdaForm$MH) ~[?:?]
at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder) ~[?:?]
at java.lang.invoke.LambdaForm$MH/0x00000001007cc040.guard(java/lang/invoke/LambdaForm$MH) ~[?:?]
at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder) ~[?:?]
at E_3a_.elastic_stack.logstash_minus_7_dot_15_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_0_dot_2_minus_java.lib.logstash.outputs.elasticsearch.http_client.bulk(E:/elastic_stack/logstash-7.15.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:137) ~[?:?]
at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder) ~[?:?]
at java.lang.invoke.LambdaForm$MH/0x0000000100856c40.invoke(java/lang/invoke/LambdaForm$MH) ~[?:?]
at java.lang.invoke.Invokers$Holder.invokeExact_MT(java/lang/invoke/Invokers$Holder) ~[?:?]
at org.jruby.RubyEnumerable$EachWithIndex.call(org/jruby/RubyEnumerable.java:1202) ~[jruby-complete-9.2.19.0.jar:?]
at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1820) ~[jruby-complete-9.2.19.0.jar:?]
at org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen) ~[jruby-complete-9.2.19.0.jar:?]
at org.jruby.RubyClass.finvokeWithRefinements(org/jruby/RubyClass.java:514) ~[jruby-complete-9.2.19.0.jar:?]