# OutOfMemory error using fsriver? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 13, 2013, 1:00am UTC](https://discuss.elastic.co/t/outofmemory-error-using-fsriver/13581 "2013-09-13T01:00:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ajitpals](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajitpals/32/2105_2.png) [@ajitpals](https://discuss.elastic.co/u/ajitpals)\ **Post date:** [September 13, 2013, 1:00am UTC](https://discuss.elastic.co/t/outofmemory-error-using-fsriver/13581/1 "2013-09-13T01:00:41Z")
Hello All, I'm parsing text file using fsriver. Recently i have a txt file of 1.2gb, when fsriver tried to read the data, it throws following error in logs. java.lang.OutOfMemoryError: Java heap space Dumping heap to java\_pid3230.hprof ... Heap dump file created [550996745 bytes in 16.871 secs] Exception in thread "elasticsearch[NC ESserver][fs\_slurper][T#1]" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2271) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.indexFile(FsRiver.java:1226) at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.addFilesRecursively(FsRiver.java:720) at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.IndexFileSystem(FsRiver.java:649) at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.run(FsRiver.java:415) at java.lang.Thread.run(Thread.java:724) I have tried changing setting for JVM and ES but nothing works. 1: using commandline = _bin/elasticsearch -f -Xmx2g -Xms2g -Des.index.storage.type=memory_ 2: tried setting JVM using : \*set JAVA\_OPTS=-Xmx2g -Xms2g \*how to solve the following problem? Thanks, Ajit -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 13, 2013, 3:19am UTC](https://discuss.elastic.co/t/outofmemory-error-using-fsriver/13581/2 "2013-09-13T03:19:11Z")
I think you need to allocate more memory to your JVM. The river needs to load the file, create a base64 version, send it to es, which could be the same node the river is running on. I suspect that you need more than 3 times memory than your biggest file. That said, I'm wondering if I could Optimize something in fsriver. Could you open an issue in fsriver project? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 13 sept. 2013 à 03:00, Ajitpal Singh [ajit.kamboj@gmail.com](mailto:ajit.kamboj@gmail.com) a écrit : Hello All, I'm parsing text file using fsriver. Recently i have a txt file of 1.2gb, when fsriver tried to read the data, it throws following error in logs. java.lang.OutOfMemoryError: Java heap space Dumping heap to java\_pid3230.hprof ... Heap dump file created [550996745 bytes in 16.871 secs] Exception in thread "elasticsearch[NC ESserver][fs\_slurper][T#1]" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2271) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.indexFile(FsRiver.java:1226) at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.addFilesRecursively(FsRiver.java:720) at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.IndexFileSystem(FsRiver.java:649) at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.run(FsRiver.java:415) at java.lang.Thread.run(Thread.java:724) I have tried changing setting for JVM and ES but nothing works. 1: using commandline = bin/elasticsearch -f -Xmx2g -Xms2g -Des.index.storage.type=memory 2: tried setting JVM using : set JAVA\_OPTS=-Xmx2g -Xms2g how to solve the following problem? Thanks, Ajit -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ajitpals](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajitpals/32/2105_2.png) [@ajitpals](https://discuss.elastic.co/u/ajitpals)\ **Post date:** [September 14, 2013, 7:08am UTC](https://discuss.elastic.co/t/outofmemory-error-using-fsriver/13581/3 "2013-09-14T07:08:02Z")
How can we create a base64 version of file and parse using fsriver? Can we have multi-part in indexing,[just an foolish thought :}] suppose we have file sample.txt (2gb) and we assume if index size \> 20mb (assumption) , then send data to different shards with following naming sample\_1.txt have first 20 mb of index and so on... I think, if we have some kind of parent mapping which include Multi-part inside, could do the work. So when we query , it return data for multi-part inside parent mapping or join the result of two different sub parts. thanks Ajit On Friday, 13 September 2013 04:19:11 UTC+1, David Pilato wrote: > I think you need to allocate more memory to your JVM. > The river needs to load the file, create a base64 version, send it to es, > which could be the same node the river is running on. I suspect that you > need more than 3 times memory than your biggest file. > > That said, I'm wondering if I could Optimize something in fsriver. Could > you open an issue in fsriver project? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 13 sept. 2013 à 03:00, Ajitpal Singh \<[ajit....@gmail.com](mailto:ajit....@gmail.com) \\> > a écrit : > > Hello All, > > I'm parsing text file using fsriver. Recently i have a txt file of 1.2gb, > when fsriver tried to read the data, it throws following error in logs. > > java.lang.OutOfMemoryError: Java heap space > Dumping heap to java\_pid3230.hprof ... > Heap dump file created [550996745 bytes in 16.871 secs] > Exception in thread "elasticsearch[NC ESserver][fs\_slurper][T#1]" > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2271) > at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) > at > java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) > at > fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.indexFile(FsRiver.java:1226) > at > fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.addFilesRecursively(FsRiver.java:720) > at > fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.IndexFileSystem(FsRiver.java:649) > at > fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.run(FsRiver.java:415) > at java.lang.Thread.run(Thread.java:724) > > I have tried changing setting for JVM and ES but nothing works. > 1: using commandline = _bin/elasticsearch -f -Xmx2g -Xms2g > -Des.index.storage.type=memory_ > > 2: tried setting JVM using : \*set JAVA\_OPTS=-Xmx2g -Xms2g > > \*how to solve the following problem? > > Thanks, > Ajit > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 15, 2013, 8:06pm UTC](https://discuss.elastic.co/t/outofmemory-error-using-fsriver/13581/4 "2013-09-15T20:06:26Z")
You don't need FSRiver to index binary content. Have a look at this project: [GitHub - elastic/elasticsearch-mapper-attachments: Mapper Attachments Type plugin for Elasticsearch](https://github.com/elasticsearch/elasticsearch-mapper-attachments) About how to encode in Base64, here is how I do it in FSRiver: [https://github.com/dadoonet/fsriver/blob/master/src/main/java/fr/pilato/elasticsearch/river/fs/river/FsRiver.java#L672](https://github.com/dadoonet/fsriver/blob/master/src/main/java/fr/pilato/elasticsearch/river/fs/river/FsRiver.java#L672) About splitting in multipart, fsriver and mapper attachment, I'm afraid that it's not supported. You could I think reduce bulk\_size to 1. It will help to hold in memory less documents and perhaps you will be able to index your big document. HTH -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 14 sept. 2013 à 09:08, Ajitpal Singh [ajit.kamboj@gmail.com](mailto:ajit.kamboj@gmail.com) a écrit : > How can we create a base64 version of file and parse using fsriver? > > Can we have multi-part in indexing,[just an foolish thought :}] suppose we have file sample.txt (2gb) and we assume > if index size \> 20mb (assumption) , then send data to different shards with following naming sample\_1.txt have first 20 mb of index and so on... > > I think, if we have some kind of parent mapping which include Multi-part inside, could do the work. So when we query , it return data for multi-part inside parent mapping or join the result of two different sub parts. > > thanks > Ajit > > On Friday, 13 September 2013 04:19:11 UTC+1, David Pilato wrote: > I think you need to allocate more memory to your JVM. > The river needs to load the file, create a base64 version, send it to es, which could be the same node the river is running on. I suspect that you need more than 3 times memory than your biggest file. > > That said, I'm wondering if I could Optimize something in fsriver. Could you open an issue in fsriver project? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 13 sept. 2013 à 03:00, Ajitpal Singh [ajit....@gmail.com](mailto:ajit....@gmail.com) a écrit : > > Hello All, > > I'm parsing text file using fsriver. Recently i have a txt file of 1.2gb, when fsriver tried to read the data, it throws following error in logs. > > java.lang.OutOfMemoryError: Java heap space > Dumping heap to java\_pid3230.hprof ... > Heap dump file created [550996745 bytes in 16.871 secs] > Exception in thread "elasticsearch[NC ESserver][fs\_slurper][T#1]" java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2271) > at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) > at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) > at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.indexFile(FsRiver.java:1226) > at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.addFilesRecursively(FsRiver.java:720) > at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.IndexFileSystem(FsRiver.java:649) > at fr.pilato.elasticsearch.river.fs.river.FsRiver$FSParser.run(FsRiver.java:415) > at java.lang.Thread.run(Thread.java:724) > > I have tried changing setting for JVM and ES but nothing works. > 1: using commandline = bin/elasticsearch -f -Xmx2g -Xms2g -Des.index.storage.type=memory > > 2: tried setting JVM using : set JAVA\_OPTS=-Xmx2g -Xms2g > > how to solve the following problem? > > Thanks, > Ajit > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:16am UTC](https://discuss.elastic.co/t/outofmemory-error-using-fsriver/13581/5 "2017-07-06T02:16:25Z")