# BulkProcessor close function **URL:** \ **Category:** Elasticsearch\ **Created:** [April 22, 2014, 9:03pm UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135 "2014-04-22T21:03:10Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![IronMike](https://avatars.discourse-cdn.com/v4/letter/i/e9a140/32.png) [@IronMike](https://discuss.elastic.co/u/IronMike)\ **Post date:** [April 22, 2014, 9:03pm UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/1 "2014-04-22T21:03:10Z")
bulkProcessor.close(); Does this also close the transportClient? When I follow up the above line with client.close(), I get no available exception. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 22, 2014, 9:42pm UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/2 "2014-04-22T21:42:23Z")
No, BulkProcessor does not close the client. Jörg On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 [sabdalla80@gmail.com](mailto:sabdalla80@gmail.com) wrote: > bulkProcessor.close(); > > Does this also close the transportClient? When I follow up the above line > with client.close(), I get no available exception. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFyhW8\_8AuztKgZG9j814rD9pgNVw%3DGQUTc-Y%2B9XvBwgQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFyhW8_8AuztKgZG9j814rD9pgNVw%3DGQUTc-Y%2B9XvBwgQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![IronMike](https://avatars.discourse-cdn.com/v4/letter/i/e9a140/32.png) [@IronMike](https://discuss.elastic.co/u/IronMike)\ **Post date:** [April 22, 2014, 9:59pm UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/3 "2014-04-22T21:59:01Z")
Ok, so how do I close the client? I have something like this: main(){ MyIndexer indexer(); for (doc=0; doc \ No, BulkProcessor does not close the client. > > Jörg > > On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 \<[sabda...@gmail.com](mailto:sabda...@gmail.com)\ > > > wrote: > > > bulkProcessor.close(); > > > > Does this also close the transportClient? When I follow up the above line > > with client.close(), I get no available exception. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/70253f01-fb50-46fe-aa9b-0336d5b70d1c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/70253f01-fb50-46fe-aa9b-0336d5b70d1c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![IronMike](https://avatars.discourse-cdn.com/v4/letter/i/e9a140/32.png) [@IronMike](https://discuss.elastic.co/u/IronMike)\ **Post date:** [April 22, 2014, 10:01pm UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/4 "2014-04-22T22:01:51Z")
Ok, so how do I close the client? I have something like this? main(){ MyIndexer indexer(); for (doc=0; doc \ No, BulkProcessor does not close the client. > > Jörg > > On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 \<[sabda...@gmail.com](mailto:sabda...@gmail.com)\ > > > wrote: > > > bulkProcessor.close(); > > > > Does this also close the transportClient? When I follow up the above line > > with client.close(), I get no available exception. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 23, 2014, 9:51am UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/5 "2014-04-23T09:51:50Z")
You must flush the BulkProcessor and wait until your code has received all responses from outstanding requests. The close() method in BulkProcessor performs a flush - and recently an explicit flush() is also implemented - > > > There is no explicit method \`flush/execute\` in \[BulkProcessor\](http://javadoc.ky…ubu.de/elasticsearch/v0.90.0/org/elasticsearch/action/bulk/BulkProcessor.html) class. This can be useful in certain scenarios. Currently it requires to close and create a new BulkProcessor if one wants an immediate flush. but BulkProcessor does not wait for all responses. If you close your client prematurely, you will see "no node available" or other things. Since a "flush" is also executed on a periodical interval, you can either simply wait in your code for an estimated time so all the bulk requests should be processed (which assumes the client is up for a long time after BulkProcessor close), or you can write a BulkProcessor wrapper that is maintaining a counter on the listener calls of "beforeBulk"/"afterBulk" listener methods so you know when all responses have been received. I wrote an improved BulkProcessor which provides a "waitForResponses" method: [https://github.com/jprante/elasticsearch-support/blob/master/src/main/java/org/xbib/elasticsearch/action/ingest/IngestProcessor.java](https://github.com/jprante/elasticsearch-support/blob/master/src/main/java/org/xbib/elasticsearch/action/ingest/IngestProcessor.java) Jörg On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 [sabdalla80@gmail.com](mailto:sabdalla80@gmail.com) wrote: > Ok, so how do I close the client? I have something like this? > > main(){ > > MyIndexer indexer(); > > for (doc=0; doc \ indexer.pushDocumentToBulk(doc); > > indexer.Shutdown(); > > } > > class MyIndexer(){ > > //create client > > // MyBulkProcesor myBulk; > > //PushDocumentToBulk(Doc) > > void ShutDown(){ > myBulk.Close(); > [//client.close](https://client.close)() ; //This creates No node available exception here, my > guess bulk is still processing ? > } > > } > > class MyBulkProcessor(){ > > BlulkProcessor bulk; > > @BeforeBulk ... > @AfterBulk .... > > PushDocumentToBulk() { ..push doc ..} > void Close(){ bulk.close(); } > } > > On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote: > > > No, BulkProcessor does not close the client. > > > > Jörg > > > > On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 [sabda...@gmail.com](mailto:sabda...@gmail.com) wrote: > > > > > bulkProcessor.close(); > > > > > > Does this also close the transportClient? When I follow up the above > > > line with client.close(), I get no available exception. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7% > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFXCV23bzyz%2BQ3mgS\_LM86K4fCJ7WGY1YKL%3DbWVJ\_PEyg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFXCV23bzyz%2BQ3mgS_LM86K4fCJ7WGY1YKL%3DbWVJ_PEyg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Preston](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_preston/32/1626_2.png) [@Matt\_Preston](https://discuss.elastic.co/u/Matt_Preston)\ **Post date:** [April 23, 2014, 10:58am UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/6 "2014-04-23T10:58:55Z")
There is an open issue to add a blocking close method to BulkProcessor > > > Blocks until all bulk requests have completed. This fixes #4158 Matt On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote: > You must flush the BulkProcessor and wait until your code has received all > responses from outstanding requests. The close() method in BulkProcessor > performs a flush - and recently an explicit flush() is also implemented - > > [[JAVA-API] BulkProcessor flush/execute · Issue #5570 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5570) > > but BulkProcessor does not wait for all responses. If you close your > client prematurely, you will see "no node available" or other things. > > Since a "flush" is also executed on a periodical interval, you can either > simply wait in your code for an estimated time so all the bulk requests > should be processed (which assumes the client is up for a long time after > BulkProcessor close), or you can write a BulkProcessor wrapper that is > maintaining a counter on the listener calls of "beforeBulk"/"afterBulk" > listener methods so you know when all responses have been received. > > I wrote an improved BulkProcessor which provides a "waitForResponses" > method: > > [https://github.com/jprante/elasticsearch-support/blob/master/src/main/java/org/xbib/elasticsearch/action/ingest/IngestProcessor.java](https://github.com/jprante/elasticsearch-support/blob/master/src/main/java/org/xbib/elasticsearch/action/ingest/IngestProcessor.java) > > Jörg > > On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 \<[sabda...@gmail.com](mailto:sabda...@gmail.com)\ > > > wrote: > > > Ok, so how do I close the client? I have something like this? > > > > main(){ > > > > MyIndexer indexer(); > > > > for (doc=0; doc \ > indexer.pushDocumentToBulk(doc); > > > > indexer.Shutdown(); > > > > } > > > > class MyIndexer(){ > > > > //create client > > > > // MyBulkProcesor myBulk; > > > > //PushDocumentToBulk(Doc) > > > > void ShutDown(){ > > myBulk.Close(); > > [//client.close](https://client.close)() ; //This creates No node available exception here, my > > guess bulk is still processing ? > > } > > > > } > > > > class MyBulkProcessor(){ > > > > BlulkProcessor bulk; > > > > @BeforeBulk ... > > @AfterBulk .... > > > > PushDocumentToBulk() { ..push doc ..} > > void Close(){ bulk.close(); } > > } > > > > On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote: > > > > > No, BulkProcessor does not close the client. > > > > > > Jörg > > > > > > On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 [sabda...@gmail.com](mailto:sabda...@gmail.com)wrote: > > > > > > > bulkProcessor.close(); > > > > > > > > Does this also close the transportClient? When I follow up the above > > > > line with client.close(), I get no available exception. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7% > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 23, 2014, 12:47pm UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/7 "2014-04-23T12:47:20Z")
Oh, thanks for the pointer. Have you tried the blocking close in case of bulk failures? It looks like it could hang forever... Personally I would prefer a separate waitForResponses() method on the semaphore without synchronization, but that is a matter of taste. Jörg On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston \< [matthew.preston@thomsonreuters.com](mailto:matthew.preston@thomsonreuters.com)\> wrote: > There is an open issue to add a blocking close method to BulkProcessor > > [Implementation of blocking close method by matt-preston · Pull Request #4180 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/4180) > > Matt > > On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote: > > > You must flush the BulkProcessor and wait until your code has received > > all responses from outstanding requests. The close() method in > > BulkProcessor performs a flush - and recently an explicit flush() is also > > implemented - > > > > [[JAVA-API] BulkProcessor flush/execute · Issue #5570 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5570) > > > > but BulkProcessor does not wait for all responses. If you close your > > client prematurely, you will see "no node available" or other things. > > > > Since a "flush" is also executed on a periodical interval, you can either > > simply wait in your code for an estimated time so all the bulk requests > > should be processed (which assumes the client is up for a long time after > > BulkProcessor close), or you can write a BulkProcessor wrapper that is > > maintaining a counter on the listener calls of "beforeBulk"/"afterBulk" > > listener methods so you know when all responses have been received. > > > > I wrote an improved BulkProcessor which provides a "waitForResponses" > > method: > > > > [https://github.com/jprante/elasticsearch-support/blob/](https://github.com/jprante/elasticsearch-support/blob/) > > master/src/main/java/org/xbib/elasticsearch/action/ingest/ > > IngestProcessor.java > > > > Jörg > > > > On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 [sabda...@gmail.com](mailto:sabda...@gmail.com) wrote: > > > > > Ok, so how do I close the client? I have something like this? > > > > > > main(){ > > > > > > MyIndexer indexer(); > > > > > > for (doc=0; doc \ > > indexer.pushDocumentToBulk(doc); > > > > > > indexer.Shutdown(); > > > > > > } > > > > > > class MyIndexer(){ > > > > > > //create client > > > > > > // MyBulkProcesor myBulk; > > > > > > //PushDocumentToBulk(Doc) > > > > > > void ShutDown(){ > > > myBulk.Close(); > > > [//client.close](https://client.close)() ; //This creates No node available exception here, > > > my guess bulk is still processing ? > > > } > > > > > > } > > > > > > class MyBulkProcessor(){ > > > > > > BlulkProcessor bulk; > > > > > > @BeforeBulk ... > > > @AfterBulk .... > > > > > > PushDocumentToBulk() { ..push doc ..} > > > void Close(){ bulk.close(); } > > > } > > > > > > On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote: > > > > > > > No, BulkProcessor does not close the client. > > > > > > > > Jörg > > > > > > > > On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 [sabda...@gmail.com](mailto:sabda...@gmail.com)wrote: > > > > > > > > > bulkProcessor.close(); > > > > > > > > > > Does this also close the transportClient? When I follow up the above > > > > > line with client.close(), I get no available exception. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40goo > > > > > [glegroups.com](http://glegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752% > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvQQjwBRuSU7Ke3iUp6%3D%2BQBX-LznqKrYf3yF59Rm3Dvw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvQQjwBRuSU7Ke3iUp6%3D%2BQBX-LznqKrYf3yF59Rm3Dvw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Preston](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_preston/32/1626_2.png) [@Matt\_Preston](https://discuss.elastic.co/u/Matt_Preston)\ **Post date:** [April 23, 2014, 2:21pm UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/8 "2014-04-23T14:21:13Z")
Hi, yes, I've tested it with bulk failures - it seems to work well. Internally, BulkProcessor releases semaphores for bulk failures and other exceptions that can be thrown by the client, like NoNodeAvailableException, so it shouldn't ever block forever. Matt On Wednesday, 23 April 2014 13:47:20 UTC+1, Jörg Prante wrote: > Oh, thanks for the pointer. > > Have you tried the blocking close in case of bulk failures? It looks like > it could hang forever... > > Personally I would prefer a separate waitForResponses() method on the > semaphore without synchronization, but that is a matter of taste. > > Jörg > > On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston \< > [matthew...@thomsonreuters.com](mailto:matthew...@thomsonreuters.com) \\> wrote: > > > There is an open issue to add a blocking close method to BulkProcessor > > > > [Implementation of blocking close method by matt-preston · Pull Request #4180 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/4180) > > > > Matt > > > > On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote: > > > > > You must flush the BulkProcessor and wait until your code has received > > > all responses from outstanding requests. The close() method in > > > BulkProcessor performs a flush - and recently an explicit flush() is also > > > implemented - > > > > > > [[JAVA-API] BulkProcessor flush/execute · Issue #5570 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5570) > > > > > > but BulkProcessor does not wait for all responses. If you close your > > > client prematurely, you will see "no node available" or other things. > > > > > > Since a "flush" is also executed on a periodical interval, you can > > > either simply wait in your code for an estimated time so all the bulk > > > requests should be processed (which assumes the client is up for a long > > > time after BulkProcessor close), or you can write a BulkProcessor wrapper > > > that is maintaining a counter on the listener calls of > > > "beforeBulk"/"afterBulk" listener methods so you know when all responses > > > have been received. > > > > > > I wrote an improved BulkProcessor which provides a "waitForResponses" > > > method: > > > > > > [https://github.com/jprante/elasticsearch-support/blob/](https://github.com/jprante/elasticsearch-support/blob/) > > > master/src/main/java/org/xbib/elasticsearch/action/ingest/ > > > IngestProcessor.java > > > > > > Jörg > > > > > > On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 [sabda...@gmail.com](mailto:sabda...@gmail.com)wrote: > > > > > > > Ok, so how do I close the client? I have something like this? > > > > > > > > main(){ > > > > > > > > MyIndexer indexer(); > > > > > > > > for (doc=0; doc \ > > > indexer.pushDocumentToBulk(doc); > > > > > > > > indexer.Shutdown(); > > > > > > > > } > > > > > > > > class MyIndexer(){ > > > > > > > > //create client > > > > > > > > // MyBulkProcesor myBulk; > > > > > > > > //PushDocumentToBulk(Doc) > > > > > > > > void ShutDown(){ > > > > myBulk.Close(); > > > > [//client.close](https://client.close)() ; //This creates No node available exception here, > > > > my guess bulk is still processing ? > > > > } > > > > > > > > } > > > > > > > > class MyBulkProcessor(){ > > > > > > > > BlulkProcessor bulk; > > > > > > > > @BeforeBulk ... > > > > @AfterBulk .... > > > > > > > > PushDocumentToBulk() { ..push doc ..} > > > > void Close(){ bulk.close(); } > > > > } > > > > > > > > On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote: > > > > > > > > > No, BulkProcessor does not close the client. > > > > > > > > > > Jörg > > > > > > > > > > On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 [sabda...@gmail.com](mailto:sabda...@gmail.com)wrote: > > > > > > > > > > > bulkProcessor.close(); > > > > > > > > > > > > Does this also close the transportClient? When I follow up the above > > > > > > line with client.close(), I get no available exception. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40goo > > > > > > [glegroups.com](http://glegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752% > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b637cef4-35a3-4d3a-92d9-0334189cd112%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b637cef4-35a3-4d3a-92d9-0334189cd112%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![IronMike](https://avatars.discourse-cdn.com/v4/letter/i/e9a140/32.png) [@IronMike](https://discuss.elastic.co/u/IronMike)\ **Post date:** [April 23, 2014, 4:10pm UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/9 "2014-04-23T16:10:26Z")
Matt, I see the open issue, but where is the code for this? On Wednesday, April 23, 2014 10:21:13 AM UTC-4, Matt Preston wrote: > Hi, > > yes, I've tested it with bulk failures - it seems to work well. > Internally, BulkProcessor releases semaphores for bulk failures and other > exceptions that can be thrown by the client, like NoNodeAvailableException, > so it shouldn't ever block forever. > > Matt > > On Wednesday, 23 April 2014 13:47:20 UTC+1, Jörg Prante wrote: > > > Oh, thanks for the pointer. > > > > Have you tried the blocking close in case of bulk failures? It looks like > > it could hang forever... > > > > Personally I would prefer a separate waitForResponses() method on the > > semaphore without synchronization, but that is a matter of taste. > > > > Jörg > > > > On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston \< > > [matthew...@thomsonreuters.com](mailto:matthew...@thomsonreuters.com)\> wrote: > > > > > There is an open issue to add a blocking close method to BulkProcessor > > > > > > [Implementation of blocking close method by matt-preston · Pull Request #4180 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/4180) > > > > > > Matt > > > > > > On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote: > > > > > > > You must flush the BulkProcessor and wait until your code has received > > > > all responses from outstanding requests. The close() method in > > > > BulkProcessor performs a flush - and recently an explicit flush() is also > > > > implemented - > > > > > > > > [[JAVA-API] BulkProcessor flush/execute · Issue #5570 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5570) > > > > > > > > but BulkProcessor does not wait for all responses. If you close your > > > > client prematurely, you will see "no node available" or other things. > > > > > > > > Since a "flush" is also executed on a periodical interval, you can > > > > either simply wait in your code for an estimated time so all the bulk > > > > requests should be processed (which assumes the client is up for a long > > > > time after BulkProcessor close), or you can write a BulkProcessor wrapper > > > > that is maintaining a counter on the listener calls of > > > > "beforeBulk"/"afterBulk" listener methods so you know when all responses > > > > have been received. > > > > > > > > I wrote an improved BulkProcessor which provides a "waitForResponses" > > > > method: > > > > > > > > [https://github.com/jprante/elasticsearch-support/blob/](https://github.com/jprante/elasticsearch-support/blob/) > > > > master/src/main/java/org/xbib/elasticsearch/action/ingest/ > > > > IngestProcessor.java > > > > > > > > Jörg > > > > > > > > On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 [sabda...@gmail.com](mailto:sabda...@gmail.com)wrote: > > > > > > > > > Ok, so how do I close the client? I have something like this? > > > > > > > > > > main(){ > > > > > > > > > > MyIndexer indexer(); > > > > > > > > > > for (doc=0; doc \ > > > > indexer.pushDocumentToBulk(doc); > > > > > > > > > > indexer.Shutdown(); > > > > > > > > > > } > > > > > > > > > > class MyIndexer(){ > > > > > > > > > > //create client > > > > > > > > > > // MyBulkProcesor myBulk; > > > > > > > > > > //PushDocumentToBulk(Doc) > > > > > > > > > > void ShutDown(){ > > > > > myBulk.Close(); > > > > > [//client.close](https://client.close)() ; //This creates No node available exception here, > > > > > my guess bulk is still processing ? > > > > > } > > > > > > > > > > } > > > > > > > > > > class MyBulkProcessor(){ > > > > > > > > > > BlulkProcessor bulk; > > > > > > > > > > @BeforeBulk ... > > > > > @AfterBulk .... > > > > > > > > > > PushDocumentToBulk() { ..push doc ..} > > > > > void Close(){ bulk.close(); } > > > > > } > > > > > > > > > > On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote: > > > > > > > > > > > No, BulkProcessor does not close the client. > > > > > > > > > > > > Jörg > > > > > > > > > > > > On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 [sabda...@gmail.com](mailto:sabda...@gmail.com)wrote: > > > > > > > > > > > > > bulkProcessor.close(); > > > > > > > > > > > > > > Does this also close the transportClient? When I follow up the above > > > > > > > line with client.close(), I get no available exception. > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/a923766f-1f7](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f7) > > > > > > > 0-4e78-8b0f-f443e16a2cd7%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752% > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c331a55c-69e4-4538-b63f-aee8fc3f9375%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c331a55c-69e4-4538-b63f-aee8fc3f9375%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:34am UTC](https://discuss.elastic.co/t/bulkprocessor-close-function/17135/10 "2017-07-06T01:34:02Z")