# What's the most efficient indexing for a river? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 13, 2013, 9:11am UTC](https://discuss.elastic.co/t/whats-the-most-efficient-indexing-for-a-river/14379 "2013-11-13T09:11:27Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\ **Post date:** [November 13, 2013, 9:11am UTC](https://discuss.elastic.co/t/whats-the-most-efficient-indexing-for-a-river/14379/1 "2013-11-13T09:11:27Z")
A question first: Are rivers in-process wrt. indexing? Or do they need to cross an HTTP boundary to do the indexing? (If the answer to this question is that indexing from a river always will have to cross an HTTP boundary, then the answer to the question in the Subject, is "bulk") Existing rivers like the RSS river, and the streaming JSON river, use bulk indexing, so I suspect the answer is that bulk is the most efficient...? I'm writing a simple river where I'm doing HTTP GET to get chunks of documents. The body if the GET response is a JSON array, with one JSON object per line, ie. something like this: [{"a": 1, "b": "c", "d": 3.14}, {"a": 2, "b": "lala", "d": 2.78}, {"a": 3, "b": "fdsf", "d": 6.023e+23}] If the river is in-process and communicates in-process with the index, then what will probably(\*) be most efficient will be to parse as little as possible and allocate as little as possible, ie. read in each line, strip leading and trailing brackets, commas and whitespace. Or alternatively, if it's possible, pull the tokens from an XContentParser and put them directly into an XContentBuilder (I haven't checked to see what's possible for this kind). Thanks! - Steinar (\*) Only actual measurement can tell, of course -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![javanna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javanna/32/4698_2.png) [@javanna](https://discuss.elastic.co/u/javanna)\ **Post date:** [November 13, 2013, 1:51pm UTC](https://discuss.elastic.co/t/whats-the-most-efficient-indexing-for-a-river/14379/2 "2013-11-13T13:51:26Z")
A river uses the Java API. Its requests don't go through the rest layer, but are already in binary format and get sent through the transport layer, port 9300 by default (tcp). Also, the client obtained within a river is a client that points to the node where the river is running. Thus your client is cluster-aware (like the node client using the Java API, even outside of elasticsearch, but this one most likely holds data too), which means that it knows where the shards and where the requests need to go. Bulk is always best as it would try and group the requests per shard and minimize the network roundtrips. On Wednesday, November 13, 2013 10:11:27 AM UTC+1, Steinar Bang wrote: > A question first: Are rivers in-process wrt. indexing? Or do they need > to cross an HTTP boundary to do the indexing? > > (If the answer to this question is that indexing from a river always > will have to cross an HTTP boundary, then the answer to the question in > the Subject, is "bulk") > > Existing rivers like the RSS river, and the streaming JSON river, use > bulk indexing, so I suspect the answer is that bulk is the most > efficient...? > > I'm writing a simple river where I'm doing HTTP GET to get chunks of > documents. The body if the GET response is a JSON array, with one JSON > object per line, ie. something like this: > [{"a": 1, "b": "c", "d": 3.14}, > {"a": 2, "b": "lala", "d": 2.78}, > {"a": 3, "b": "fdsf", "d": 6.023e+23}] > > If the river is in-process and communicates in-process with the index, > then what will probably(\*) be most efficient will be to parse as little > as possible and allocate as little as possible, ie. read in each line, > strip leading and trailing brackets, commas and whitespace. > > Or alternatively, if it's possible, pull the tokens from an > XContentParser and put them directly into an XContentBuilder (I haven't > checked to see what's possible for this kind). > > Thanks! > > - Steinar > > (\*) Only actual measurement can tell, of course -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\ **Post date:** [November 13, 2013, 5:28pm UTC](https://discuss.elastic.co/t/whats-the-most-efficient-indexing-for-a-river/14379/3 "2013-11-13T17:28:03Z")
> > > > > Luca Cavanna [cavannaluca@gmail.com](mailto:cavannaluca@gmail.com): > A river uses the Java API. Its requests don't go through the rest > layer, but are already in binary format and get sent through the > transport layer, port 9300 by default (tcp). Ah, ok. Somewhere in between the two possibilities I envisioned...🙂 > Also, the client obtained within a river is a client that points to > the node where the river is running. Thus your client is cluster-aware > (like the node client using the Java API, even outside of > elasticsearch, but this one most likely holds data too), which means > that it knows where the shards and where the requests need to go. Bulk > is always best as it would try and group the requests per shard and > minimize the network roundtrips. Yes, as long as there is a network crossing involved, then bulk will undoubtedly be the best. Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:07am UTC](https://discuss.elastic.co/t/whats-the-most-efficient-indexing-for-a-river/14379/4 "2017-07-06T02:07:30Z")
Closed on July 6, 2017, 2:07am UTC