# Logstash throws java.lang.OutOfMemoryError: Java heap space no matter the heap size **URL:** \ **Category:** Logstash\ **Created:** [April 16, 2023, 6:00pm UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089 "2023-04-16T18:00:31Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![ste1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ste1/32/119876_2.png) [@ste1](https://discuss.elastic.co/u/ste1)\ **Post date:** [April 16, 2023, 6:00pm UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089/1 "2023-04-16T18:00:31Z")
Im attempting to parse a huge (few million lines) csv file with logstash and output it to elasticsearch. ```auto [FATAL] 2023-04-16 19:00:19.011 [LogStash::Runner] Logstash - java.lang.OutOfMemoryError: Java heap space at java.nio.HeapCharBuffer.(java/nio/HeapCharBuffer.java:61) ~[?:?] at java.nio.CharBuffer.allocate(java/nio/CharBuffer.java:348) ~[?:?] at java.nio.charset.CharsetDecoder.decode(java/nio/charset/CharsetDecoder.java:807) ~[?:?] at java.nio.charset.Charset.decode(java/nio/charset/Charset.java:814) ~[?:?] at org.jruby.RubyEncoding.decodeUTF8(org/jruby/RubyEncoding.java:297) ~[jruby-complete-9.2.20.1.jar:?] at org.jruby.RubyString.decodeString(org/jruby/RubyString.java:802) ~[jruby-complete-9.2.20.1.jar:?] at org.jruby.RubyString.toString(org/jruby/RubyString.java:793) ~[jruby-complete-9.2.20.1.jar:?] at org.logstash.Javafier.lambda$initConverters$1(org/logstash/Javafier.java:88) ~[logstash-core.jar:?] at org.logstash.Javafier$$Lambda$586/0x00000001012b4c40.convert(org/logstash/Javafier$$Lambda$586/0x00000001012b4c40) ~[?:?] at org.logstash.Javafier.deep(org/logstash/Javafier.java:57) ~[logstash-core.jar:?] at org.logstash.Event.getField(org/logstash/Event.java:177) ~[logstash-core.jar:?] at org.logstash.StringInterpolation.evaluate(org/logstash/StringInterpolation.java:86) ~[logstash-core.jar:?] at org.logstash.Event.sprintf(org/logstash/Event.java:363) ~[logstash-core.jar:?] at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.sprintf(org/logstash/ext/JrubyEventExtLibrary.java:202) ~[logstash-core.jar:?] at java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java/lang/invoke/DirectMethodHandle$Holder) ~[?:?] at java.lang.invoke.LambdaForm$MH/0x0000000100780840.invoke(java/lang/invoke/LambdaForm$MH) ~[?:?] at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder) ~[?:?] at java.lang.invoke.LambdaForm$MH/0x0000000100737c40.guard(java/lang/invoke/LambdaForm$MH) ~[?:?] at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder) ~[?:?] at java.lang.invoke.LambdaForm$MH/0x0000000100737c40.guard(java/lang/invoke/LambdaForm$MH) ~[?:?] at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder) ~[?:?] at usr.share.logstash.logstash_minus_core.lib.logstash.util.decorators.add_fields(/usr/share/logstash/logstash-core/lib/logstash/util/decorators.rb:34) ~[?:?] at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder) ~[?:?] at java.lang.invoke.LambdaForm$MH/0x00000001012b7840.invoke(java/lang/invoke/LambdaForm$MH) ~[?:?] at java.lang.invoke.Invokers$Holder.invokeExact_MT(java/lang/invoke/Invokers$Holder) ~[?:?] at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1821) ~[jruby-complete-9.2.20.1.jar:?] at java.lang.invoke.LambdaForm$DMH/0x0000000100763040.invokeVirtual(java/lang/invoke/LambdaForm$DMH) ~[?:?] at java.lang.invoke.LambdaForm$MH/0x0000000100780840.invoke(java/lang/invoke/LambdaForm$MH) ~[?:?] { ``` is thrown after a few minutes of running logstash. There are similar threads asking about this very question but none are properly answered. Inside my /etc/logstash/jvm.options: -Xms2g -Xmx2g my machine has 8gb of ram. I have tried lowering the heap memory, I have tried making it higher but nothing helps. If I put it too high after a while logstash will just crash with "Killed". /etc/elasticsearch/jvm.options.d/jvmheap.options has: -Xms2g -Xmx2g I dont know what to do as I really cannot afford purchasing a server with more RAM but need to parse this file no matter what. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 16, 2023, 6:35pm UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089/2 "2023-04-16T18:35:31Z")
What does your configuration look like? Inputs, filters... ---
**Author:** ![ste1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ste1/32/119876_2.png) [@ste1](https://discuss.elastic.co/u/ste1)\ **Post date:** [April 16, 2023, 6:38pm UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089/3 "2023-04-16T18:38:41Z")
```auto input { file { path => "path/to/file.csv" start_position => "beginning" } } filter { csv { autodetect_column_names => false columns => ["username", "uid"] target => "_tmp" } mutate { add_field => { "[data][username]" => "%{[_tmp][username]}" "[data][uid]" => "%{[_tmp][uid]}" } } mutate { remove_field => ["_tmp"] } prune { whitelist_names => ["data"] } } output { elasticsearch { hosts => ["http://localhost:9200/"] index => "uids" } stdout{} } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 16, 2023, 7:18pm UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089/4 "2023-04-16T19:18:09Z")
I would not expect that configuration to need more than 300 MB to run in! You can simplify the filters a little ``` csv { columns => ["username", "uid"] target => "data" autogenerate_column_names => false } ``` will only parse the first two columns of the CSV file, so you can remove the mutate filters and even the prune. The OOM is happening in the add\_field, although I doubt removing it will change much. Reducing pipeline.batch.size from the default of 125 might help. ---
**Author:** ![ste1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ste1/32/119876_2.png) [@ste1](https://discuss.elastic.co/u/ste1)\ **Post date:** [April 16, 2023, 7:30pm UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089/5 "2023-04-16T19:30:38Z")
There are multiple columns in the CSV file and im extracting only the ones I need, unfortunately getting the first two columns would not work Edit: Decreasing pipeline.batch.size did not help :^( ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 16, 2023, 8:05pm UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089/6 "2023-04-16T20:05:21Z")
> [@ste1](#): > > unfortunately getting the first two columns would not work But that is what your filter configuration would do if it didn't run out of memory! The first two columns will be named username and uid, the rest will be column3, column4, etc, and will get remove when you delete [\_tmp] ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [April 17, 2023, 10:57am UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089/7 "2023-04-17T10:57:36Z")
> [@Badger](#): > > Reducing pipeline.batch.size from the default of 125 might help. How about `pipeline.workers` to 1? Will be slower, however will use less resources. Also, maybe `file_chunk_count`, default value is `4611686018427387903`, to reduce reading all at once. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 15, 2023, 10:58am UTC](https://discuss.elastic.co/t/logstash-throws-java-lang-outofmemoryerror-java-heap-space-no-matter-the-heap-size/330089/8 "2023-05-15T10:58:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.