# Does the server support streaming? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2014, 4:58pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148 "2014-01-08T16:58:34Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Ryan\_Pedela](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_pedela/32/27156_2.png) [@Ryan\_Pedela](https://discuss.elastic.co/u/Ryan_Pedela)\ **Post date:** [January 8, 2014, 4:58pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/1 "2014-01-08T16:58:34Z")
Let's say I have a million documents I want to index. I am aware that you can index 100 documents at a time or 1000 at a time using the bulk API. However I could also write my HTTP client to stream all one million documents as bytes with a single bulk API call. This would be advantageous because the number of round-trips would be reduced which would reduce HTTP overhead. In addition, I will have predictable memory usage on the client. In a batch, there could be documents that are a few KB, a few MB, or a few GB. This means it can be hard to predict the memory requirements of the client. With byte streaming, the memory requirements are whatever I set the buffer to. My question: does the ES server support streaming a large API call? Or does it store the entire API call in memory and then process it? I have the same question for search results but in reverse. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6e494a74-3ffb-4afb-88af-61f2fb799eee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6e494a74-3ffb-4afb-88af-61f2fb799eee%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 8, 2014, 6:02pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/2 "2014-01-08T18:02:25Z")
You are correct, ES nodes consumes data request by request, before they are passed on through the cluster. Also the bulk indexing requests, such requests are temporarily pushed to buffers, but they are split by lines and executed as single actions. So to reduce network roundtrips, the best thing is to use the bulk API. What is left is a few percent to optimize, which is not much worth it. With gzip, ES HTTP provides transparent compression. Main challenge is HTTP overhead (headers can't be compressed), and base64, if you use binary data with ES. Please note that you must evaluate the bulk responses too, in order to validate the notification about bulk success on doc level. It is possible to extend the whole ES API also to Websocket, so beside JSON, it could also be possible to transfer JSON text frames or SMILE/binary frames on a single bi-directional channel. HTTP must use two channels for this, so with Websocket, you can reduce connection resources to the half. In this sense, the Netty channel / REST / Java API could be extended for special realtime WS streaming mode applications, like for pubsub applications. I experimented with that some time ago on ES 0.20 [https://github.com/jprante/elasticsearch-transport-websocket](https://github.com/jprante/elasticsearch-transport-websocket) (needs updating) From what I understand, the thrift transport plugin compiles the ES API, operates in a streaming-like fashion, and is providing a solution that reduces HTTP overhead: > **[elastic/elasticsearch-transport-thrift](https://github.com/elastic/elasticsearch-transport-thrift)** > > elasticsearch-transport-thrift - Thrift Transport for elasticsearch (STOPPED) Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH7wM%2BpdVpH9%3Dysoq7a0CesOGxDnY4yAwQAeAcqLWDGvQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH7wM%2BpdVpH9%3Dysoq7a0CesOGxDnY4yAwQAeAcqLWDGvQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![eunever32](https://avatars.discourse-cdn.com/v4/letter/e/e68b1a/32.png) [@eunever32](https://discuss.elastic.co/u/eunever32)\ **Post date:** [March 2, 2014, 5:46pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/3 "2014-03-02T17:46:41Z")
Hi, I have a similar question as the OP : what is the best way to get 1m or 30m records indexed? I mean I can send client.bulk batches of records but while the request is being indexed the client is waiting: valuable seconds. Also I have tried: python elasticsearch-py and there is a helpers.bulk and helpers.streaming\_bulk And looking at the source code I can see that helpers.bulk calls -\> helpers.streaming\_bulk so is it the same thing? ie I should continue to call helpers.bulk? Or what is the difference? Thanks, On Wednesday, January 8, 2014 6:02:25 PM UTC, Jörg Prante wrote: > You are correct, ES nodes consumes data request by request, before they > are passed on through the cluster. Also the bulk indexing requests, such > requests are temporarily pushed to buffers, but they are split by lines and > executed as single actions. > > So to reduce network roundtrips, the best thing is to use the bulk API. > What is left is a few percent to optimize, which is not much worth it. With > gzip, ES HTTP provides transparent compression. Main challenge is HTTP > overhead (headers can't be compressed), and base64, if you use binary data > with ES. > > Please note that you must evaluate the bulk responses too, in order to > validate the notification about bulk success on doc level. > > It is possible to extend the whole ES API also to Websocket, so beside > JSON, it could also be possible to transfer JSON text frames or > SMILE/binary frames on a single bi-directional channel. HTTP must use two > channels for this, so with Websocket, you can reduce connection resources > to the half. In this sense, the Netty channel / REST / Java API could be > extended for special realtime WS streaming mode applications, like for > pubsub applications. I experimented with that some time ago on ES 0.20 > [GitHub - jprante/elasticsearch-transport-websocket: WebSockets for ElasticSearch](https://github.com/jprante/elasticsearch-transport-websocket) (needs > updating) > > From what I understand, the thrift transport plugin compiles the ES API, > operates in a streaming-like fashion, and is providing a solution that > reduces HTTP overhead: > [GitHub - elastic/elasticsearch-transport-thrift: Thrift Transport for elasticsearch (STOPPED)](https://github.com/elasticsearch/elasticsearch-transport-thrift) > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Honza\_Kral](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/honza_kral/32/876_2.png) [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)\ **Post date:** [March 2, 2014, 5:51pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/4 "2014-03-02T17:51:15Z")
Hi, the streaming\_bulk function in elasticsearch-py is a helper that will actually split the stream of documents into chunk and send them to elasticsearch - it does not stream all documents to es as a single request. It is impossible (due to the nature of bulk requests) for elasticsearch to consume arbitrary number of documents in a single request so this helper was created to give you the abstraction. The difference between bulk and streaming\_bulk is in the way it's executed and returned - bulk will just return statistics/errors while streaming\_bulk is a generator that will keep yielding results per document, thus completely hiding the fact that the stream is being sent to ES in chunks. Hope this helps, Honza On Sun, Mar 2, 2014 at 6:46 PM, [eunever32@gmail.com](mailto:eunever32@gmail.com) wrote: > Hi, > > I have a similar question as the OP : what is the best way to get 1m or 30m > records indexed? > I mean I can send client.bulk batches of records but while the request is > being indexed the client is waiting: valuable seconds. > > Also I have tried: python elasticsearch-py and there is a helpers.bulk and > helpers.streaming\_bulk > And looking at the source code I can see that helpers.bulk calls -\> > helpers.streaming\_bulk so is it the same thing? ie > I should continue to call helpers.bulk? Or what is the difference? > > Thanks, > > On Wednesday, January 8, 2014 6:02:25 PM UTC, Jörg Prante wrote: > > > You are correct, ES nodes consumes data request by request, before they > > are passed on through the cluster. Also the bulk indexing requests, such > > requests are temporarily pushed to buffers, but they are split by lines and > > executed as single actions. > > > > So to reduce network roundtrips, the best thing is to use the bulk API. > > What is left is a few percent to optimize, which is not much worth it. With > > gzip, ES HTTP provides transparent compression. Main challenge is HTTP > > overhead (headers can't be compressed), and base64, if you use binary data > > with ES. > > > > Please note that you must evaluate the bulk responses too, in order to > > validate the notification about bulk success on doc level. > > > > It is possible to extend the whole ES API also to Websocket, so beside > > JSON, it could also be possible to transfer JSON text frames or SMILE/binary > > frames on a single bi-directional channel. HTTP must use two channels for > > this, so with Websocket, you can reduce connection resources to the half. In > > this sense, the Netty channel / REST / Java API could be extended for > > special realtime WS streaming mode applications, like for pubsub > > applications. I experimented with that some time ago on ES 0.20 > > [GitHub - jprante/elasticsearch-transport-websocket: WebSockets for ElasticSearch](https://github.com/jprante/elasticsearch-transport-websocket) (needs > > updating) > > > > From what I understand, the thrift transport plugin compiles the ES API, > > operates in a streaming-like fashion, and is providing a solution that > > reduces HTTP overhead: > > [GitHub - elastic/elasticsearch-transport-thrift: Thrift Transport for elasticsearch (STOPPED)](https://github.com/elasticsearch/elasticsearch-transport-thrift) > > > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABfdDipwE%2BEXrNEnn00%2BdM0ci\_mSZektwKEf3Bpv6v3ydEd1pA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABfdDipwE%2BEXrNEnn00%2BdM0ci_mSZektwKEf3Bpv6v3ydEd1pA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![eunever32](https://avatars.discourse-cdn.com/v4/letter/e/e68b1a/32.png) [@eunever32](https://discuss.elastic.co/u/eunever32)\ **Post date:** [March 2, 2014, 6:17pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/5 "2014-03-02T18:17:11Z")
Hey thanks, So is there a convenient way to asynchronously call the bulk (helpers bulk or helpers streaming\_bulk) in a way that means the client isn't waiting for the request to complete? On Sunday, March 2, 2014 5:51:15 PM UTC, Honza Král wrote: > Hi, > > the streaming\_bulk function in elasticsearch-py is a helper that will > actually split the stream of documents into chunk and send them to > elasticsearch - it does not stream all documents to es as a single > request. It is impossible (due to the nature of bulk requests) for > elasticsearch to consume arbitrary number of documents in a single > request so this helper was created to give you the abstraction. > > The difference between bulk and streaming\_bulk is in the way it's > executed and returned - bulk will just return statistics/errors while > streaming\_bulk is a generator that will keep yielding results per > document, thus completely hiding the fact that the stream is being > sent to ES in chunks. > > Hope this helps, > Honza > > On Sun, Mar 2, 2014 at 6:46 PM, \<[eune...@gmail.com](mailto:eune...@gmail.com) \\> wrote: > > > Hi, > > > > I have a similar question as the OP : what is the best way to get 1m or > > 30m > > records indexed? > > I mean I can send client.bulk batches of records but while the request > > is > > being indexed the client is waiting: valuable seconds. > > > > Also I have tried: python elasticsearch-py and there is a helpers.bulk > > and > > helpers.streaming\_bulk > > And looking at the source code I can see that helpers.bulk calls -\> > > helpers.streaming\_bulk so is it the same thing? ie > > I should continue to call helpers.bulk? Or what is the difference? > > > > Thanks, > > > > On Wednesday, January 8, 2014 6:02:25 PM UTC, Jörg Prante wrote: > > > > > You are correct, ES nodes consumes data request by request, before they > > > are passed on through the cluster. Also the bulk indexing requests, > > > such > > > requests are temporarily pushed to buffers, but they are split by lines > > > and > > > executed as single actions. > > > > > > So to reduce network roundtrips, the best thing is to use the bulk API. > > > What is left is a few percent to optimize, which is not much worth it. > > > With > > > gzip, ES HTTP provides transparent compression. Main challenge is HTTP > > > overhead (headers can't be compressed), and base64, if you use binary > > > data > > > with ES. > > > > > > Please note that you must evaluate the bulk responses too, in order to > > > validate the notification about bulk success on doc level. > > > > > > It is possible to extend the whole ES API also to Websocket, so beside > > > JSON, it could also be possible to transfer JSON text frames or > > > SMILE/binary > > > frames on a single bi-directional channel. HTTP must use two channels > > > for > > > this, so with Websocket, you can reduce connection resources to the > > > half. In > > > this sense, the Netty channel / REST / Java API could be extended for > > > special realtime WS streaming mode applications, like for pubsub > > > applications. I experimented with that some time ago on ES 0.20 > > > [GitHub - jprante/elasticsearch-transport-websocket: WebSockets for ElasticSearch](https://github.com/jprante/elasticsearch-transport-websocket) (needs > > > updating) > > > > > > From what I understand, the thrift transport plugin compiles the ES > > > API, > > > operates in a streaming-like fashion, and is providing a solution that > > > reduces HTTP overhead: > > > [GitHub - elastic/elasticsearch-transport-thrift: Thrift Transport for elasticsearch (STOPPED)](https://github.com/elasticsearch/elasticsearch-transport-thrift) > > > > > > Jörg > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f0bc0dc9-7341-4eb4-9e78-ff5200a03635%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0bc0dc9-7341-4eb4-9e78-ff5200a03635%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Honza\_Kral](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/honza_kral/32/876_2.png) [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)\ **Post date:** [March 2, 2014, 6:21pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/6 "2014-03-02T18:21:14Z")
Well, you can just use any async http library to do it, but I wouldn't recommend it since putting it all back together might be difficult (to see which documents failed to index etc). You can always just have a couple of threads each running a streaming bulk, reading from a Queue and writing the results to another Queue, should be fairly easy to do in your code. On Sun, Mar 2, 2014 at 7:17 PM, [eunever32@gmail.com](mailto:eunever32@gmail.com) wrote: > Hey thanks, > > So is there a convenient way to asynchronously call the bulk (helpers bulk > or helpers streaming\_bulk) in a way that means the client isn't waiting for > the request to complete? > > On Sunday, March 2, 2014 5:51:15 PM UTC, Honza Král wrote: > > > Hi, > > > > the streaming\_bulk function in elasticsearch-py is a helper that will > > actually split the stream of documents into chunk and send them to > > elasticsearch - it does not stream all documents to es as a single > > request. It is impossible (due to the nature of bulk requests) for > > elasticsearch to consume arbitrary number of documents in a single > > request so this helper was created to give you the abstraction. > > > > The difference between bulk and streaming\_bulk is in the way it's > > executed and returned - bulk will just return statistics/errors while > > streaming\_bulk is a generator that will keep yielding results per > > document, thus completely hiding the fact that the stream is being > > sent to ES in chunks. > > > > Hope this helps, > > Honza > > > > On Sun, Mar 2, 2014 at 6:46 PM, [eune...@gmail.com](mailto:eune...@gmail.com) wrote: > > > > > Hi, > > > > > > I have a similar question as the OP : what is the best way to get 1m or > > > 30m > > > records indexed? > > > I mean I can send client.bulk batches of records but while the request > > > is > > > being indexed the client is waiting: valuable seconds. > > > > > > Also I have tried: python elasticsearch-py and there is a helpers.bulk > > > and > > > helpers.streaming\_bulk > > > And looking at the source code I can see that helpers.bulk calls -\> > > > helpers.streaming\_bulk so is it the same thing? ie > > > I should continue to call helpers.bulk? Or what is the difference? > > > > > > Thanks, > > > > > > On Wednesday, January 8, 2014 6:02:25 PM UTC, Jörg Prante wrote: > > > > > > > You are correct, ES nodes consumes data request by request, before they > > > > are passed on through the cluster. Also the bulk indexing requests, > > > > such > > > > requests are temporarily pushed to buffers, but they are split by lines > > > > and > > > > executed as single actions. > > > > > > > > So to reduce network roundtrips, the best thing is to use the bulk API. > > > > What is left is a few percent to optimize, which is not much worth it. > > > > With > > > > gzip, ES HTTP provides transparent compression. Main challenge is HTTP > > > > overhead (headers can't be compressed), and base64, if you use binary > > > > data > > > > with ES. > > > > > > > > Please note that you must evaluate the bulk responses too, in order to > > > > validate the notification about bulk success on doc level. > > > > > > > > It is possible to extend the whole ES API also to Websocket, so beside > > > > JSON, it could also be possible to transfer JSON text frames or > > > > SMILE/binary > > > > frames on a single bi-directional channel. HTTP must use two channels > > > > for > > > > this, so with Websocket, you can reduce connection resources to the > > > > half. In > > > > this sense, the Netty channel / REST / Java API could be extended for > > > > special realtime WS streaming mode applications, like for pubsub > > > > applications. I experimented with that some time ago on ES 0.20 > > > > [GitHub - jprante/elasticsearch-transport-websocket: WebSockets for ElasticSearch](https://github.com/jprante/elasticsearch-transport-websocket) (needs > > > > updating) > > > > > > > > From what I understand, the thrift transport plugin compiles the ES > > > > API, > > > > operates in a streaming-like fashion, and is providing a solution that > > > > reduces HTTP overhead: > > > > [GitHub - elastic/elasticsearch-transport-thrift: Thrift Transport for elasticsearch (STOPPED)](https://github.com/elasticsearch/elasticsearch-transport-thrift) > > > > > > > > Jörg > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/f0bc0dc9-7341-4eb4-9e78-ff5200a03635%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0bc0dc9-7341-4eb4-9e78-ff5200a03635%40googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABfdDir2qbnprp6hbnv4paB8S9GdGEwDbjnhPnqFxX%3DxSn9SOg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABfdDir2qbnprp6hbnv4paB8S9GdGEwDbjnhPnqFxX%3DxSn9SOg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![eunever32](https://avatars.discourse-cdn.com/v4/letter/e/e68b1a/32.png) [@eunever32](https://discuss.elastic.co/u/eunever32)\ **Post date:** [March 2, 2014, 7:22pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/7 "2014-03-02T19:22:54Z")
so excuse me Honza.. am I correct thinking there is no point from a performance perspective calling helpers.bulk because it will just be "sliced" into the chunk size by streaming bulk anyway. It would make more sense to call helpers.streaming\_bulk directly to reduce the client side activity? And actually from a raw performance perspective just call client.bulk ? Thanks On Sunday, March 2, 2014 6:21:14 PM UTC, Honza Král wrote: > Well, you can just use any async http library to do it, but I wouldn't > recommend it since putting it all back together might be difficult (to > see which documents failed to index etc). You can always just have a > couple of threads each running a streaming bulk, reading from a Queue > and writing the results to another Queue, should be fairly easy to do > in your code. > > On Sun, Mar 2, 2014 at 7:17 PM, \<[eune...@gmail.com](mailto:eune...@gmail.com) \\> wrote: > > > Hey thanks, > > > > So is there a convenient way to asynchronously call the bulk (helpers > > bulk > > or helpers streaming\_bulk) in a way that means the client isn't waiting > > for > > the request to complete? > > > > On Sunday, March 2, 2014 5:51:15 PM UTC, Honza Král wrote: > > > > > Hi, > > > > > > the streaming\_bulk function in elasticsearch-py is a helper that will > > > actually split the stream of documents into chunk and send them to > > > elasticsearch - it does not stream all documents to es as a single > > > request. It is impossible (due to the nature of bulk requests) for > > > elasticsearch to consume arbitrary number of documents in a single > > > request so this helper was created to give you the abstraction. > > > > > > The difference between bulk and streaming\_bulk is in the way it's > > > executed and returned - bulk will just return statistics/errors while > > > streaming\_bulk is a generator that will keep yielding results per > > > document, thus completely hiding the fact that the stream is being > > > sent to ES in chunks. > > > > > > Hope this helps, > > > Honza > > > > > > On Sun, Mar 2, 2014 at 6:46 PM, [eune...@gmail.com](mailto:eune...@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > I have a similar question as the OP : what is the best way to get 1m > > > > or > > > > 30m > > > > records indexed? > > > > I mean I can send client.bulk batches of records but while the > > > > request > > > > is > > > > being indexed the client is waiting: valuable seconds. > > > > > > > > Also I have tried: python elasticsearch-py and there is a > > > > helpers.bulk > > > > and > > > > helpers.streaming\_bulk > > > > And looking at the source code I can see that helpers.bulk calls -\> > > > > helpers.streaming\_bulk so is it the same thing? ie > > > > I should continue to call helpers.bulk? Or what is the difference? > > > > > > > > Thanks, > > > > > > > > On Wednesday, January 8, 2014 6:02:25 PM UTC, Jörg Prante wrote: > > > > > > > > > You are correct, ES nodes consumes data request by request, before > > > > > they > > > > > are passed on through the cluster. Also the bulk indexing requests, > > > > > such > > > > > requests are temporarily pushed to buffers, but they are split by > > > > > lines > > > > > and > > > > > executed as single actions. > > > > > > > > > > So to reduce network roundtrips, the best thing is to use the bulk > > > > > API. > > > > > What is left is a few percent to optimize, which is not much worth > > > > > it. > > > > > With > > > > > gzip, ES HTTP provides transparent compression. Main challenge is > > > > > HTTP > > > > > overhead (headers can't be compressed), and base64, if you use > > > > > binary > > > > > data > > > > > with ES. > > > > > > > > > > Please note that you must evaluate the bulk responses too, in order > > > > > to > > > > > validate the notification about bulk success on doc level. > > > > > > > > > > It is possible to extend the whole ES API also to Websocket, so > > > > > beside > > > > > JSON, it could also be possible to transfer JSON text frames or > > > > > SMILE/binary > > > > > frames on a single bi-directional channel. HTTP must use two > > > > > channels > > > > > for > > > > > this, so with Websocket, you can reduce connection resources to the > > > > > half. In > > > > > this sense, the Netty channel / REST / Java API could be extended > > > > > for > > > > > special realtime WS streaming mode applications, like for pubsub > > > > > applications. I experimented with that some time ago on ES 0.20 > > > > > [GitHub - jprante/elasticsearch-transport-websocket: WebSockets for ElasticSearch](https://github.com/jprante/elasticsearch-transport-websocket) (needs > > > > > updating) > > > > > > > > > > From what I understand, the thrift transport plugin compiles the ES > > > > > API, > > > > > operates in a streaming-like fashion, and is providing a solution > > > > > that > > > > > reduces HTTP overhead: > > > > > [GitHub - elastic/elasticsearch-transport-thrift: Thrift Transport for elasticsearch (STOPPED)](https://github.com/elasticsearch/elasticsearch-transport-thrift) > > > > > > > > > > Jörg > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, > > > > send > > > > an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/f0bc0dc9-7341-4eb4-9e78-ff5200a03635%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0bc0dc9-7341-4eb4-9e78-ff5200a03635%40googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8ae39c83-3805-40d8-8299-298578353e59%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8ae39c83-3805-40d8-8299-298578353e59%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Honza\_Kral](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/honza_kral/32/876_2.png) [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)\ **Post date:** [March 2, 2014, 7:29pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/8 "2014-03-02T19:29:12Z")
The difference between stream\_bulk and regular bulk is just in the API, under the hood they both perform the same operation. The only difference is that bulk only returns once all documents have been sent whereas streaming\_bulk is a generator that keeps yielding individual results. On Sun, Mar 2, 2014 at 8:22 PM, [eunever32@gmail.com](mailto:eunever32@gmail.com) wrote: > so excuse me Honza.. am I correct thinking there is no point from a > performance perspective calling helpers.bulk because it will just be > "sliced" into the chunk size by streaming bulk anyway. > > It would make more sense to call helpers.streaming\_bulk directly to reduce > the client side activity? > > And actually from a raw performance perspective just call client.bulk ? > > Thanks > > On Sunday, March 2, 2014 6:21:14 PM UTC, Honza Král wrote: > > > Well, you can just use any async http library to do it, but I wouldn't > > recommend it since putting it all back together might be difficult (to > > see which documents failed to index etc). You can always just have a > > couple of threads each running a streaming bulk, reading from a Queue > > and writing the results to another Queue, should be fairly easy to do > > in your code. > > > > On Sun, Mar 2, 2014 at 7:17 PM, [eune...@gmail.com](mailto:eune...@gmail.com) wrote: > > > > > Hey thanks, > > > > > > So is there a convenient way to asynchronously call the bulk (helpers > > > bulk > > > or helpers streaming\_bulk) in a way that means the client isn't waiting > > > for > > > the request to complete? > > > > > > On Sunday, March 2, 2014 5:51:15 PM UTC, Honza Král wrote: > > > > > > > Hi, > > > > > > > > the streaming\_bulk function in elasticsearch-py is a helper that will > > > > actually split the stream of documents into chunk and send them to > > > > elasticsearch - it does not stream all documents to es as a single > > > > request. It is impossible (due to the nature of bulk requests) for > > > > elasticsearch to consume arbitrary number of documents in a single > > > > request so this helper was created to give you the abstraction. > > > > > > > > The difference between bulk and streaming\_bulk is in the way it's > > > > executed and returned - bulk will just return statistics/errors while > > > > streaming\_bulk is a generator that will keep yielding results per > > > > document, thus completely hiding the fact that the stream is being > > > > sent to ES in chunks. > > > > > > > > Hope this helps, > > > > Honza > > > > > > > > On Sun, Mar 2, 2014 at 6:46 PM, [eune...@gmail.com](mailto:eune...@gmail.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > I have a similar question as the OP : what is the best way to get 1m > > > > > or > > > > > 30m > > > > > records indexed? > > > > > I mean I can send client.bulk batches of records but while the > > > > > request > > > > > is > > > > > being indexed the client is waiting: valuable seconds. > > > > > > > > > > Also I have tried: python elasticsearch-py and there is a > > > > > helpers.bulk > > > > > and > > > > > helpers.streaming\_bulk > > > > > And looking at the source code I can see that helpers.bulk calls -\> > > > > > helpers.streaming\_bulk so is it the same thing? ie > > > > > I should continue to call helpers.bulk? Or what is the difference? > > > > > > > > > > Thanks, > > > > > > > > > > On Wednesday, January 8, 2014 6:02:25 PM UTC, Jörg Prante wrote: > > > > > > > > > > > You are correct, ES nodes consumes data request by request, before > > > > > > they > > > > > > are passed on through the cluster. Also the bulk indexing requests, > > > > > > such > > > > > > requests are temporarily pushed to buffers, but they are split by > > > > > > lines > > > > > > and > > > > > > executed as single actions. > > > > > > > > > > > > So to reduce network roundtrips, the best thing is to use the bulk > > > > > > API. > > > > > > What is left is a few percent to optimize, which is not much worth > > > > > > it. > > > > > > With > > > > > > gzip, ES HTTP provides transparent compression. Main challenge is > > > > > > HTTP > > > > > > overhead (headers can't be compressed), and base64, if you use > > > > > > binary > > > > > > data > > > > > > with ES. > > > > > > > > > > > > Please note that you must evaluate the bulk responses too, in order > > > > > > to > > > > > > validate the notification about bulk success on doc level. > > > > > > > > > > > > It is possible to extend the whole ES API also to Websocket, so > > > > > > beside > > > > > > JSON, it could also be possible to transfer JSON text frames or > > > > > > SMILE/binary > > > > > > frames on a single bi-directional channel. HTTP must use two > > > > > > channels > > > > > > for > > > > > > this, so with Websocket, you can reduce connection resources to the > > > > > > half. In > > > > > > this sense, the Netty channel / REST / Java API could be extended > > > > > > for > > > > > > special realtime WS streaming mode applications, like for pubsub > > > > > > applications. I experimented with that some time ago on ES 0.20 > > > > > > [GitHub - jprante/elasticsearch-transport-websocket: WebSockets for ElasticSearch](https://github.com/jprante/elasticsearch-transport-websocket) (needs > > > > > > updating) > > > > > > > > > > > > From what I understand, the thrift transport plugin compiles the ES > > > > > > API, > > > > > > operates in a streaming-like fashion, and is providing a solution > > > > > > that > > > > > > reduces HTTP overhead: > > > > > > [GitHub - elastic/elasticsearch-transport-thrift: Thrift Transport for elasticsearch (STOPPED)](https://github.com/elasticsearch/elasticsearch-transport-thrift) > > > > > > > > > > > > Jörg > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups > > > > > "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > send > > > > > an > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b2702386-ca31-4551-9a92-15775a9011d2%40googlegroups.com). > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/f0bc0dc9-7341-4eb4-9e78-ff5200a03635%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0bc0dc9-7341-4eb4-9e78-ff5200a03635%40googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8ae39c83-3805-40d8-8299-298578353e59%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8ae39c83-3805-40d8-8299-298578353e59%40googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABfdDiqxXZ%2Be4uT\_Fmv8WZE1HjUd94LM9Y6wVnQD\_j1Gp6MbEg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABfdDiqxXZ%2Be4uT_Fmv8WZE1HjUd94LM9Y6wVnQD_j1Gp6MbEg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Randall\_McRee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/randall_mcree/32/47177_2.png) [@Randall\_McRee](https://discuss.elastic.co/u/Randall_McRee)\ **Post date:** [March 3, 2014, 11:28pm UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/9 "2014-03-03T23:28:45Z")
HTTP overhead is miniscule compared to the server-side (elasticsearch) resources which are required to index the documents. Even with bulk and no streaming etc. the bottleneck is in building the index and in particular, disk I/O (primarily) as well as cpu and memory. So, regardless, your client will always end up waiting for the server to finish these necessary tasks since they require so much more time and space than simply sending the documents across the network. On Wed, Jan 8, 2014 at 8:58 AM, Ryan Pedela [rpedela@datalanche.com](mailto:rpedela@datalanche.com) wrote: > Let's say I have a million documents I want to index. I am aware that you > can index 100 documents at a time or 1000 at a time using the bulk API. > However I could also write my HTTP client to stream all one million > documents as bytes with a single bulk API call. This would be advantageous > because the number of round-trips would be reduced which would reduce HTTP > overhead. In addition, I will have predictable memory usage on the client. > In a batch, there could be documents that are a few KB, a few MB, or a few > GB. This means it can be hard to predict the memory requirements of the > client. With byte streaming, the memory requirements are whatever I set the > buffer to. > > My question: does the ES server support streaming a large API call? Or > does it store the entire API call in memory and then process it? I have the > same question for search results but in reverse. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/6e494a74-3ffb-4afb-88af-61f2fb799eee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6e494a74-3ffb-4afb-88af-61f2fb799eee%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAFjHw360\_3FZm0\_-GoRCbsb-9ZtuBuyf8%3Dqr0g6PsMUTWt41%3DA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAFjHw360_3FZm0_-GoRCbsb-9ZtuBuyf8%3Dqr0g6PsMUTWt41%3DA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [March 12, 2014, 3:27am UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/10 "2014-03-12T03:27:26Z")
Hi, The key is to find the ideal bulk size and the ideal bulk request concurrency level, and then make sure the client always feeds ES enough data to achieve (close to) ideal utilization and minimize idling on either side. ## Otis Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Monday, March 3, 2014 6:28:45 PM UTC-5, RKM wrote: > HTTP overhead is miniscule compared to the server-side (elasticsearch) > resources which are required to index the documents. Even with bulk and no > streaming etc. the bottleneck is in building the index and in particular, > disk I/O (primarily) as well as cpu and memory. > > So, regardless, your client will always end up waiting for the server to > finish these necessary tasks since they require so much more time and space > than simply sending the documents across the network. > > On Wed, Jan 8, 2014 at 8:58 AM, Ryan Pedela \<[rpe...@datalanche.com](mailto:rpe...@datalanche.com)\ > > > wrote: > > > Let's say I have a million documents I want to index. I am aware that you > > can index 100 documents at a time or 1000 at a time using the bulk API. > > However I could also write my HTTP client to stream all one million > > documents as bytes with a single bulk API call. This would be advantageous > > because the number of round-trips would be reduced which would reduce HTTP > > overhead. In addition, I will have predictable memory usage on the client. > > In a batch, there could be documents that are a few KB, a few MB, or a few > > GB. This means it can be hard to predict the memory requirements of the > > client. With byte streaming, the memory requirements are whatever I set the > > buffer to. > > > > My question: does the ES server support streaming a large API call? Or > > does it store the entire API call in memory and then process it? I have the > > same question for search results but in reverse. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/6e494a74-3ffb-4afb-88af-61f2fb799eee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6e494a74-3ffb-4afb-88af-61f2fb799eee%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d31c868e-d12b-4340-a047-9c3d2eb673a6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d31c868e-d12b-4340-a047-9c3d2eb673a6%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:44am UTC](https://discuss.elastic.co/t/does-the-server-support-streaming/15148/11 "2017-07-06T01:44:01Z")