# Running out of memory when parsing the large text file **URL:** \ **Category:** Elasticsearch\ **Created:** [January 10, 2014, 1:59am UTC](https://discuss.elastic.co/t/running-out-of-memory-when-parsing-the-large-text-file/15188 "2014-01-10T01:59:13Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Ivan\_Ji](https://avatars.discourse-cdn.com/v4/letter/i/d78d45/32.png) [@Ivan\_Ji](https://discuss.elastic.co/u/Ivan_Ji)\ **Post date:** [January 10, 2014, 1:59am UTC](https://discuss.elastic.co/t/running-out-of-memory-when-parsing-the-large-text-file/15188/1 "2014-01-10T01:59:13Z")
Hi all, I post several large text files, which are about 20~30MB and contains all the text, into ES. And I use the attachment mapper to be the field type to store these file. It cost memory very much. Even when I post one file, the used memory grows from about 150MB to 250MB. BTW, I use the default tokenizer for these field. Although this file can be generated many tokens, but what I don't understand is the memory cost. Does it store all the tokens into memory? Ideas? Cheers, Ivan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2f200f67-7024-4cdd-9c68-05875f0155ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2f200f67-7024-4cdd-9c68-05875f0155ca%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan\_Ji](https://avatars.discourse-cdn.com/v4/letter/i/d78d45/32.png) [@Ivan\_Ji](https://discuss.elastic.co/u/Ivan_Ji)\ **Post date:** [March 10, 2014, 7:52am UTC](https://discuss.elastic.co/t/running-out-of-memory-when-parsing-the-large-text-file/15188/2 "2014-03-10T07:52:06Z")
Hi all, This time, I read the string content out from the file and push it into a field inside a document, whose analyzer is standard. Without the attachment mapper, the same condition occurred. It occur " java.lang.OutOfMemoryError: Java heap spac" when the total index is just 400MB and the document count is 10. What's the suggestions for these large text file ? I am considering using more smart analyzer which might eliminate some redundancies. but are there any other ? cheers, Ivan Ivan Ji於 2014年1月10日星期五UTC+8上午9時59分13秒寫道： > Hi all, > > I post several large text files, which are about 20~30MB and contains all > the text, into ES. And I use the attachment mapper to be the field type to > store these file. > It cost memory very much. Even when I post one file, the used memory grows > from about 150MB to 250MB. BTW, I use the default tokenizer for these field. > > Although this file can be generated many tokens, but what I don't > understand is the memory cost. Does it store all the tokens into memory? > > Ideas? > > Cheers, > > Ivan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/75fa9adc-03c8-45a4-9360-c0926a7d0f1d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/75fa9adc-03c8-45a4-9360-c0926a7d0f1d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:44am UTC](https://discuss.elastic.co/t/running-out-of-memory-when-parsing-the-large-text-file/15188/3 "2017-07-06T01:44:34Z")
Closed on July 6, 2017, 1:44am UTC