# \[Java\] Stream large file while indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [February 14, 2012, 3:15pm UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706 "2012-02-14T15:15:41Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [February 14, 2012, 3:15pm UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/1 "2012-02-14T15:15:41Z")
Hi , While indexing i have a field whose value is quite large. This value is stored in a file as text. I prefer not to load the entire file into main memory to read the content. Instead i would prefer this file is streamed directly to ES without affecting much of RAM. Is this possible with java API of ES ? Thanks Vineeth ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [February 14, 2012, 8:55pm UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/2 "2012-02-14T20:55:19Z")
I don't believe this is possible. Would need some soft of JSON streaming support. How big are you talking about? If you're not storing as its own stored field or part of the \_source JSON, probably should be OK search side. Thanks, Paul On Feb 14, 8:15 am, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > Hi , > > While indexing i have a field whose value is quite large. > This value is stored in a file as text. > > I prefer not to load the entire file into main memory to read the content. > Instead i would prefer this file is streamed directly to ES without > affecting much of RAM. > > Is this possible with java API of ES ? > > Thanks > Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [February 15, 2012, 8:33am UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/3 "2012-02-15T08:33:08Z")
Its not on search side. Mine is a multi threaded application and it is trying to index tonns of a file at a time to ES. So am trying to access main memory as les as possible. Thanks Vineeth On Wed, Feb 15, 2012 at 2:25 AM, ppearcy [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > I don't believe this is possible. Would need some soft of JSON > streaming support. > > How big are you talking about? If you're not storing as its own stored > field or part of the \_source JSON, probably should be OK search side. > > Thanks, > Paul > > On Feb 14, 8:15 am, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > > > Hi , > > > > While indexing i have a field whose value is quite large. > > This value is stored in a file as text. > > > > I prefer not to load the entire file into main memory to read the > > content. > > Instead i would prefer this file is streamed directly to ES without > > affecting much of RAM. > > > > Is this possible with java API of ES ? > > > > Thanks > > Vineeth ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 15, 2012, 10:39am UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/4 "2012-02-15T10:39:16Z")
There is no support for streaming. On Wednesday, February 15, 2012 at 10:33 AM, Vineeth Mohan wrote: > Its not on search side. > Mine is a multi threaded application and it is trying to index tonns of a file at a time to ES. > So am trying to access main memory as les as possible. > > Thanks > Vineeth > > On Wed, Feb 15, 2012 at 2:25 AM, ppearcy \<[ppearcy@gmail.com](mailto:ppearcy@gmail.com) ([mailto:ppearcy@gmail.com](mailto:ppearcy@gmail.com))\> wrote: > > > I don't believe this is possible. Would need some soft of JSON > > streaming support. > > > > How big are you talking about? If you're not storing as its own stored > > field or part of the \_source JSON, probably should be OK search side. > > > > Thanks, > > Paul > > > > On Feb 14, 8:15 am, Vineeth Mohan \<[vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) ([mailto:vineethmo...@algotree.com](mailto:vineethmo...@algotree.com))\> wrote: > > > > > Hi , > > > > > > While indexing i have a field whose value is quite large. > > > This value is stored in a file as text. > > > > > > I prefer not to load the entire file into main memory to read the content. > > > Instead i would prefer this file is streamed directly to ES without > > > affecting much of RAM. > > > > > > Is this possible with java API of ES ? > > > > > > Thanks > > > Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [February 17, 2012, 12:03pm UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/5 "2012-02-17T12:03:32Z")
Hello Shay , Can you give clues or hints on how to implement this over the existing API ?? I was hoping to accomplish this task by overriding couple of methods. Any pointers in this direction would be appreciated. Thanks Vineeth On Wed, Feb 15, 2012 at 4:09 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > There is no support for streaming. > > On Wednesday, February 15, 2012 at 10:33 AM, Vineeth Mohan wrote: > > Its not on search side. > Mine is a multi threaded application and it is trying to index tonns of a > file at a time to ES. > So am trying to access main memory as les as possible. > > Thanks > Vineeth > > On Wed, Feb 15, 2012 at 2:25 AM, ppearcy [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > > I don't believe this is possible. Would need some soft of JSON > streaming support. > > How big are you talking about? If you're not storing as its own stored > field or part of the \_source JSON, probably should be OK search side. > > Thanks, > Paul > > On Feb 14, 8:15 am, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > > > Hi , > > > > While indexing i have a field whose value is quite large. > > This value is stored in a file as text. > > > > I prefer not to load the entire file into main memory to read the > > content. > > Instead i would prefer this file is streamed directly to ES without > > affecting much of RAM. > > > > Is this possible with java API of ES ? > > > > Thanks > > Vineeth ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 17, 2012, 5:48pm UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/6 "2012-02-17T17:48:46Z")
Not really, there is no support for streaming single doc large data. On Friday, February 17, 2012 at 2:03 PM, Vineeth Mohan wrote: > Hello Shay , > > Can you give clues or hints on how to implement this over the existing API ?? > I was hoping to accomplish this task by overriding couple of methods. > > Any pointers in this direction would be appreciated. > > Thanks > Vineeth > > On Wed, Feb 15, 2012 at 4:09 PM, Shay Banon \<[kimchy@gmail.com](mailto:kimchy@gmail.com) ([mailto:kimchy@gmail.com](mailto:kimchy@gmail.com))\> wrote: > > > There is no support for streaming. > > > > On Wednesday, February 15, 2012 at 10:33 AM, Vineeth Mohan wrote: > > > > > Its not on search side. > > > Mine is a multi threaded application and it is trying to index tonns of a file at a time to ES. > > > So am trying to access main memory as les as possible. > > > > > > Thanks > > > Vineeth > > > > > > On Wed, Feb 15, 2012 at 2:25 AM, ppearcy \<[ppearcy@gmail.com](mailto:ppearcy@gmail.com) ([mailto:ppearcy@gmail.com](mailto:ppearcy@gmail.com))\> wrote: > > > > > > > I don't believe this is possible. Would need some soft of JSON > > > > streaming support. > > > > > > > > How big are you talking about? If you're not storing as its own stored > > > > field or part of the \_source JSON, probably should be OK search side. > > > > > > > > Thanks, > > > > Paul > > > > > > > > On Feb 14, 8:15 am, Vineeth Mohan \<[vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) ([mailto:vineethmo...@algotree.com](mailto:vineethmo...@algotree.com))\> wrote: > > > > > > > > > Hi , > > > > > > > > > > While indexing i have a field whose value is quite large. > > > > > This value is stored in a file as text. > > > > > > > > > > I prefer not to load the entire file into main memory to read the content. > > > > > Instead i would prefer this file is streamed directly to ES without > > > > > affecting much of RAM. > > > > > > > > > > Is this possible with java API of ES ? > > > > > > > > > > Thanks > > > > > Vineeth ---
**Author:** ![Yermakovich\_Siarhei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yermakovich_siarhei/32/2361_2.png) [@Yermakovich\_Siarhei](https://discuss.elastic.co/u/Yermakovich_Siarhei)\ **Post date:** [May 6, 2013, 7:34pm UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/7 "2013-05-06T19:34:25Z")
Hi Shay, Has anything changed since? Maybe some third-party plugins exist, that allow to do large files streaming to Elasticsearch? On Friday, February 17, 2012 8:48:46 PM UTC+3, kimchy wrote: > Not really, there is no support for streaming single doc large data. > > On Friday, February 17, 2012 at 2:03 PM, Vineeth Mohan wrote: > > Hello Shay , > > Can you give clues or hints on how to implement this over the existing API > ?? > I was hoping to accomplish this task by overriding couple of methods. > > Any pointers in this direction would be appreciated. > > Thanks > Vineeth > > On Wed, Feb 15, 2012 at 4:09 PM, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com)\ > > > wrote: > > There is no support for streaming. > > On Wednesday, February 15, 2012 at 10:33 AM, Vineeth Mohan wrote: > > Its not on search side. > Mine is a multi threaded application and it is trying to index tonns of a > file at a time to ES. > So am trying to access main memory as les as possible. > > Thanks > Vineeth > > On Wed, Feb 15, 2012 at 2:25 AM, ppearcy \<[ppe...@gmail.com](mailto:ppe...@gmail.com) \\>wrote: > > I don't believe this is possible. Would need some soft of JSON > streaming support. > > How big are you talking about? If you're not storing as its own stored > field or part of the \_source JSON, probably should be OK search side. > > Thanks, > Paul > > On Feb 14, 8:15 am, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > > > Hi , > > > > While indexing i have a field whose value is quite large. > > This value is stored in a file as text. > > > > I prefer not to load the entire file into main memory to read the > > content. > > Instead i would prefer this file is streamed directly to ES without > > affecting much of RAM. > > > > Is this possible with java API of ES ? > > > > Thanks > > Vineeth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [May 7, 2013, 12:38am UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/8 "2013-05-07T00:38:10Z")
There are several different aspects when it comes to streaming files to Elasticsearch. 1. The first is, input format is JSON, and the indexer should create large JSON docs, but for tiny display later (highlighting). 2. The second is, you want binary files store inside Lucene unindexed (for whatever reason). The main challenge is how to handle this with JSON (it requires something like base64 encoding in combination with compression) 3. And the third is, you want Elasticsearch do smart Lucene index codec processing to create documents from streams "on the fly" -\> 1. I think there is no advantage in streaming for this case, since Lucene needs somewhere the whole document in memory for the inverted index statistics computation. If your input documents are large, just add enough heap memory to get them processed. I'm not sure but most search engines out there including Google have a hard limit how much of a document is analyzed for highlighting or term indexing (the first 10k characters maybe?). The reason is better performance. So maybe there is little sense to enforce these features on Elasticsearch with large docs. -\> 2. you should think about it twice and maybe it is better to store these files outside Elasticsearch. Retrieving large stored docs may hit your performance, also relocating shards will be slow. -\> 3. you can implement a custom codec for Lucene 4 that may handle your content streams gracefully. Such a codec is unfortunately domain-specific, since it depends on the nature of the stream elements. For example, such a codec could do complex event stream processing (CEP) like in Esper [http://esper.codehaus.org/](http://esper.codehaus.org/) My 2c. Jörg Am 06.05.13 21:34, schrieb Yermakovich Siarhei: > Has anything changed since? Maybe some third-party plugins exist, that > allow to do large files streaming to Elasticsearch? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Hao](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Hao](https://discuss.elastic.co/u/Hao)\ **Post date:** [March 27, 2015, 6:59pm UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/9 "2015-03-27T18:59:12Z")
Hi Vineeth, I am suffering from the same issue here(I am using .net REST API, NEST) Have you found solution of indexing large files? Thank you Best Regards Hao On Tuesday, February 14, 2012 at 3:15:41 PM UTC, Vineeth Mohan wrote: > Hi , > > While indexing i have a field whose value is quite large. > This value is stored in a file as text. > > I prefer not to load the entire file into main memory to read the content. > Instead i would prefer this file is streamed directly to ES without > affecting much of RAM. > > Is this possible with java API of ES ? > > Thanks > Vineeth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1316bb3c-4dae-4c58-9815-c36636c31c2e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1316bb3c-4dae-4c58-9815-c36636c31c2e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [March 27, 2015, 8:17pm UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/10 "2015-03-27T20:17:16Z")
I believe elasticsearch loads the whole indexed document into ram before indexing. It certainly loads the whole document in ram for things like source filtering. Lucene doesn't require this, but elasticsearch does it because for the typical use case its fine. On Mar 27, 2015 2:59 PM, "Hao" [hao.qian.career@gmail.com](mailto:hao.qian.career@gmail.com) wrote: > Hi Vineeth, > > I am suffering from the same issue here(I am using .net REST API, NEST) > Have you found solution of indexing large files? Thank you > > Best Regards > Hao > > On Tuesday, February 14, 2012 at 3:15:41 PM UTC, Vineeth Mohan wrote: > > > Hi , > > > > While indexing i have a field whose value is quite large. > > This value is stored in a file as text. > > > > I prefer not to load the entire file into main memory to read the content. > > Instead i would prefer this file is streamed directly to ES without > > affecting much of RAM. > > > > Is this possible with java API of ES ? > > > > Thanks > > Vineeth > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1316bb3c-4dae-4c58-9815-c36636c31c2e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1316bb3c-4dae-4c58-9815-c36636c31c2e%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/1316bb3c-4dae-4c58-9815-c36636c31c2e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1316bb3c-4dae-4c58-9815-c36636c31c2e%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0jQTNA43tssxrjsEjfJwf6%2Bh7Z-RGZpQHewRArB8bGHg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0jQTNA43tssxrjsEjfJwf6%2Bh7Z-RGZpQHewRArB8bGHg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:23am UTC](https://discuss.elastic.co/t/java-stream-large-file-while-indexing/6706/11 "2017-07-06T00:23:24Z")