# Using the Streamable field value in Java API **URL:** \ **Category:** Elasticsearch\ **Created:** [November 9, 2012, 8:22am UTC](https://discuss.elastic.co/t/using-the-streamable-field-value-in-java-api/9642 "2012-11-09T08:22:35Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Atharva\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atharva_patel/32/2414_2.png) [@Atharva\_Patel](https://discuss.elastic.co/u/Atharva_Patel)\ **Post date:** [November 9, 2012, 8:22am UTC](https://discuss.elastic.co/t/using-the-streamable-field-value-in-java-api/9642/1 "2012-11-09T08:22:35Z")
I have a use case where I need to append certain amount of text to an existing field of the document which has been already indexed earlier. After appending I will be reindexing the document. I am predicting the amount of data which is going to be stored in that single field of my document will be pretty large so I decided not to store the field as well as disabled the \_source feature on that document type. I also feel as there will be several indexing/reindexing operations will be going on simultaneously in the JVM (I am currently using Java API), it will be highly memory inefficient to bring all the whole field value as large string object into memory. I am wondering if the '\*streamability' \*of the field in document in Java API can be used in someway to make my use case memory efficient. If yes, I would like to see a kind of pseudocode or example for steps of stream operations to perform to achieve the this use case memory efficiently? -- ---
**Author:** ![Atharva\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atharva_patel/32/2414_2.png) [@Atharva\_Patel](https://discuss.elastic.co/u/Atharva_Patel)\ **Post date:** [November 9, 2012, 9:03am UTC](https://discuss.elastic.co/t/using-the-streamable-field-value-in-java-api/9642/2 "2012-11-09T09:03:54Z")
Can the Text, XContent, ByteReference of elasticsearch be used here to achieve this? On Friday, 9 November 2012 13:52:35 UTC+5:30, Atharva Patel wrote: > I have a use case where I need to append certain amount of text to an > existing field of the document which has been already indexed earlier. > After appending I will be reindexing the document. I am predicting the > amount of data which is going to be stored in that single field of my > document will be pretty large so I decided not to store the field as well > as disabled the \_source feature on that document type. > > I also feel as there will be several indexing/reindexing operations will > be going on simultaneously in the JVM (I am currently using Java API), it > will be highly memory inefficient to bring all the whole field value as > large string object into memory. > > I am wondering if the '\*streamability' \*of the field in document in Java > API can be used in someway to make my use case memory efficient. If yes, I > would like to see a kind of pseudocode or example for steps of stream > operations to perform to achieve the this use case memory efficiently? -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 9, 2012, 2:12pm UTC](https://discuss.elastic.co/t/using-the-streamable-field-value-in-java-api/9642/3 "2012-11-09T14:12:48Z")
Hi Atharva, you are right, streamability of data is a concern. ES nodes transport all the data over the wire and provides an API for that. But, consider when it comes to indexing a Lucene Field from a list of terms. In Lucene 3, you have to add all the terms and fields to the document, before the document got indexed. It is not possible to get an advantage by streaming field data, it is just another style of providing terms for the Lucene analysis. In Lucene 4, field data is processed by a pluggable codecs. With a custom codec that is able to consume large term streams efficiently, it could be possible to benefit from the idea to stream large term streams into a single Lucene field. But, for creating an inverted index, it is required to reference to all the document data with all the field in memory at once for generating term/field/doc statistics for relevance scoring. That is the price you pay for inverted indexing. Or your scenario is different. You do not want term indexing, but simply storing key/value like data where the value is an opaque binary stream. I think there are also enhancements available in Lucene 4 for key/value style. Another related issue is JSON combined with binary stream data. ES encodes binary streams into base64, and offers compression to remedy the overhead situation. We will wait and see what improvements ES will offer in 0.21 and later! Jörg On Friday, November 9, 2012 10:03:54 AM UTC+1, Atharva Patel wrote: > Can the Text, XContent, ByteReference of elasticsearch be used here to > achieve this? > > On Friday, 9 November 2012 13:52:35 UTC+5:30, Atharva Patel wrote: > > > I have a use case where I need to append certain amount of text to an > > existing field of the document which has been already indexed earlier. > > After appending I will be reindexing the document. I am predicting the > > amount of data which is going to be stored in that single field of my > > document will be pretty large so I decided not to store the field as well > > as disabled the \_source feature on that document type. > > > > I also feel as there will be several indexing/reindexing operations will > > be going on simultaneously in the JVM (I am currently using Java API), it > > will be highly memory inefficient to bring all the whole field value as > > large string object into memory. > > > > I am wondering if the '\*streamability' \*of the field in document in Java > > API can be used in someway to make my use case memory efficient. If yes, I > > would like to see a kind of pseudocode or example for steps of stream > > operations to perform to achieve the this use case memory efficiently? -- ---
**Author:** ![Chris\_Male](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_male/32/2607_2.png) [@Chris\_Male](https://discuss.elastic.co/u/Chris_Male)\ **Post date:** [November 10, 2012, 1:48am UTC](https://discuss.elastic.co/t/using-the-streamable-field-value-in-java-api/9642/4 "2012-11-10T01:48:38Z")
The ability to stream data into Lucene is fairly unexplored I think but in Lucene 4 there is probably enough flexibility for it now. The notion of Documents and Fields are heavily abstracted and the only requirement for indexing content is to be able to generate a TokenStream. I hope it's something we can look into more in future ES versions. On Saturday, November 10, 2012 3:12:48 AM UTC+13, Jörg Prante wrote: > Hi Atharva, > > you are right, streamability of data is a concern. ES nodes transport all > the data over the wire and provides an API for that. > > But, consider when it comes to indexing a Lucene Field from a list of > terms. In Lucene 3, you have to add all the terms and fields to the > document, before the document got indexed. It is not possible to get an > advantage by streaming field data, it is just another style of providing > terms for the Lucene analysis. > > In Lucene 4, field data is processed by a pluggable codecs. With a custom > codec that is able to consume large term streams efficiently, it could be > possible to benefit from the idea to stream large term streams into a > single Lucene field. But, for creating an inverted index, it is required to > reference to all the document data with all the field in memory at once for > generating term/field/doc statistics for relevance scoring. That is the > price you pay for inverted indexing. > > Or your scenario is different. You do not want term indexing, but simply > storing key/value like data where the value is an opaque binary stream. I > think there are also enhancements available in Lucene 4 for key/value style. > > Another related issue is JSON combined with binary stream data. ES encodes > binary streams into base64, and offers compression to remedy the overhead > situation. > > We will wait and see what improvements ES will offer in 0.21 and later! > > Jörg > > On Friday, November 9, 2012 10:03:54 AM UTC+1, Atharva Patel wrote: > > > Can the Text, XContent, ByteReference of elasticsearch be used here to > > achieve this? > > > > On Friday, 9 November 2012 13:52:35 UTC+5:30, Atharva Patel wrote: > > > > > I have a use case where I need to append certain amount of text to an > > > existing field of the document which has been already indexed earlier. > > > After appending I will be reindexing the document. I am predicting the > > > amount of data which is going to be stored in that single field of my > > > document will be pretty large so I decided not to store the field as well > > > as disabled the \_source feature on that document type. > > > > > > I also feel as there will be several indexing/reindexing operations will > > > be going on simultaneously in the JVM (I am currently using Java API), it > > > will be highly memory inefficient to bring all the whole field value as > > > large string object into memory. > > > > > > I am wondering if the '\*streamability' \*of the field in document in > > > Java API can be used in someway to make my use case memory efficient. If > > > yes, I would like to see a kind of pseudocode or example for steps of > > > stream operations to perform to achieve the this use case memory > > > efficiently? -- ---
**Author:** ![Atharva\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atharva_patel/32/2414_2.png) [@Atharva\_Patel](https://discuss.elastic.co/u/Atharva_Patel)\ **Post date:** [November 10, 2012, 8:38am UTC](https://discuss.elastic.co/t/using-the-streamable-field-value-in-java-api/9642/5 "2012-11-10T08:38:57Z")
Thanks Jorg for the detailed response. After your explanation about Lucene 3's weakness in dealing with streams, I wondered if there is some limitation on the maximum field value length in bytes. On search I found that there is an option for setting the max field length on IndexWriter with: IndexWriter.MaxFieldLength[http://lucene.apache.org/core/old\_versioned\_docs/versions/3\_0\_0/api/all/org/apache/lucene/index/IndexWriter.MaxFieldLength.html](http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/api/all/org/apache/lucene/index/IndexWriter.MaxFieldLength.html) It is possible to set it to unlimited which restricts it by Integer.MAX\_VALUE. So is there a way to specify this in ES, or whether the default itself is unlimited in ES? Does the Lucene 3's non-streamability also imply that it may raise Out Of Memory Exception while passing it large streams through ES field value? On Friday, 9 November 2012 19:42:48 UTC+5:30, Jörg Prante wrote: > Hi Atharva, > > you are right, streamability of data is a concern. ES nodes transport all > the data over the wire and provides an API for that. > > But, consider when it comes to indexing a Lucene Field from a list of > terms. In Lucene 3, you have to add all the terms and fields to the > document, before the document got indexed. It is not possible to get an > advantage by streaming field data, it is just another style of providing > terms for the Lucene analysis. > > In Lucene 4, field data is processed by a pluggable codecs. With a custom > codec that is able to consume large term streams efficiently, it could be > possible to benefit from the idea to stream large term streams into a > single Lucene field. But, for creating an inverted index, it is required to > reference to all the document data with all the field in memory at once for > generating term/field/doc statistics for relevance scoring. That is the > price you pay for inverted indexing. > > Or your scenario is different. You do not want term indexing, but simply > storing key/value like data where the value is an opaque binary stream. I > think there are also enhancements available in Lucene 4 for key/value style. > > Another related issue is JSON combined with binary stream data. ES encodes > binary streams into base64, and offers compression to remedy the overhead > situation. > > We will wait and see what improvements ES will offer in 0.21 and later! > > Jörg > > On Friday, November 9, 2012 10:03:54 AM UTC+1, Atharva Patel wrote: > > > Can the Text, XContent, ByteReference of elasticsearch be used here to > > achieve this? > > > > On Friday, 9 November 2012 13:52:35 UTC+5:30, Atharva Patel wrote: > > > > > I have a use case where I need to append certain amount of text to an > > > existing field of the document which has been already indexed earlier. > > > After appending I will be reindexing the document. I am predicting the > > > amount of data which is going to be stored in that single field of my > > > document will be pretty large so I decided not to store the field as well > > > as disabled the \_source feature on that document type. > > > > > > I also feel as there will be several indexing/reindexing operations will > > > be going on simultaneously in the JVM (I am currently using Java API), it > > > will be highly memory inefficient to bring all the whole field value as > > > large string object into memory. > > > > > > I am wondering if the '\*streamability' \*of the field in document in > > > Java API can be used in someway to make my use case memory efficient. If > > > yes, I would like to see a kind of pseudocode or example for steps of > > > stream operations to perform to achieve the this use case memory > > > efficiently? -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 11, 2012, 9:51am UTC](https://discuss.elastic.co/t/using-the-streamable-field-value-in-java-api/9642/6 "2012-11-11T09:51:38Z")
You should increase the HTTP request upload limit which is set to 100 MB by default in ES. There is no field length limit in Lucene set by ES, so I think it is Integer.MAX\_VALUE in IndexWriter which is depreceated. Chirs Male stated it, the new limit is only restricted by the numbers of tokens, which is only an option. For example, it can be set in [http://lucene.apache.org/core/3\_6\_1/api/all/org/apache/lucene/analysis/LimitTokenCountAnalyzer.html](http://lucene.apache.org/core/3_6_1/api/all/org/apache/lucene/analysis/LimitTokenCountAnalyzer.html) Of course, you will enter OOMs if the heap size of the nodes can't handle the amount of data. You need to experiment. Heap limits can be changed in bin/elasticsearch.in.sh Cheers, Jörg On Saturday, November 10, 2012 9:38:57 AM UTC+1, Atharva Patel wrote: > Thanks Jorg for the detailed response. After your explanation about Lucene > 3's weakness in dealing with streams, I wondered if there is some > limitation on the maximum field value length in bytes. On search I found > that there is an option for setting the max field length on IndexWriter > with: > IndexWriter.MaxFieldLength[http://lucene.apache.org/core/old\_versioned\_docs/versions/3\_0\_0/api/all/org/apache/lucene/index/IndexWriter.MaxFieldLength.html](http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/api/all/org/apache/lucene/index/IndexWriter.MaxFieldLength.html) It > is possible to set it to unlimited which restricts it by Integer.MAX\_VALUE. > So is there a way to specify this in ES, or whether the default itself is > unlimited in ES? > > Does the Lucene 3's non-streamability also imply that it may raise Out Of > Memory Exception while passing it large streams through ES field value? > > On Friday, 9 November 2012 19:42:48 UTC+5:30, Jörg Prante wrote: > > > Hi Atharva, > > > > you are right, streamability of data is a concern. ES nodes transport all > > the data over the wire and provides an API for that. > > > > But, consider when it comes to indexing a Lucene Field from a list of > > terms. In Lucene 3, you have to add all the terms and fields to the > > document, before the document got indexed. It is not possible to get an > > advantage by streaming field data, it is just another style of providing > > terms for the Lucene analysis. > > > > In Lucene 4, field data is processed by a pluggable codecs. With a custom > > codec that is able to consume large term streams efficiently, it could be > > possible to benefit from the idea to stream large term streams into a > > single Lucene field. But, for creating an inverted index, it is required to > > reference to all the document data with all the field in memory at once for > > generating term/field/doc statistics for relevance scoring. That is the > > price you pay for inverted indexing. > > > > Or your scenario is different. You do not want term indexing, but simply > > storing key/value like data where the value is an opaque binary stream. I > > think there are also enhancements available in Lucene 4 for key/value style. > > > > Another related issue is JSON combined with binary stream data. ES > > encodes binary streams into base64, and offers compression to remedy the > > overhead situation. > > > > We will wait and see what improvements ES will offer in 0.21 and later! > > > > Jörg > > > > On Friday, November 9, 2012 10:03:54 AM UTC+1, Atharva Patel wrote: > > > > > Can the Text, XContent, ByteReference of elasticsearch be used here to > > > achieve this? > > > > > > On Friday, 9 November 2012 13:52:35 UTC+5:30, Atharva Patel wrote: > > > > > > > I have a use case where I need to append certain amount of text to an > > > > existing field of the document which has been already indexed earlier. > > > > After appending I will be reindexing the document. I am predicting the > > > > amount of data which is going to be stored in that single field of my > > > > document will be pretty large so I decided not to store the field as well > > > > as disabled the \_source feature on that document type. > > > > > > > > I also feel as there will be several indexing/reindexing operations > > > > will be going on simultaneously in the JVM (I am currently using Java API), > > > > it will be highly memory inefficient to bring all the whole field value as > > > > large string object into memory. > > > > > > > > I am wondering if the '\*streamability' \*of the field in document in > > > > Java API can be used in someway to make my use case memory efficient. If > > > > yes, I would like to see a kind of pseudocode or example for steps of > > > > stream operations to perform to achieve the this use case memory > > > > efficiently? -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:04am UTC](https://discuss.elastic.co/t/using-the-streamable-field-value-in-java-api/9642/7 "2017-07-06T03:04:55Z")