You must flush the BulkProcessor and wait until your code has received all
responses from outstanding requests. The close() method in BulkProcessor
performs a flush - and recently an explicit flush() is also implemented -
but BulkProcessor does not wait for all responses. If you close your client
prematurely, you will see "no node available" or other things.
Since a "flush" is also executed on a periodical interval, you can either
simply wait in your code for an estimated time so all the bulk requests
should be processed (which assumes the client is up for a long time after
BulkProcessor close), or you can write a BulkProcessor wrapper that is
maintaining a counter on the listener calls of "beforeBulk"/"afterBulk"
listener methods so you know when all responses have been received.
I wrote an improved BulkProcessor which provides a "waitForResponses"
method:
Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
There is an open issue to add a blocking close method to BulkProcessor
Matt
On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:
You must flush the BulkProcessor and wait until your code has received all
responses from outstanding requests. The close() method in BulkProcessor
performs a flush - and recently an explicit flush() is also implemented -
but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see "no node available" or other things.
Since a "flush" is also executed on a periodical interval, you can either
simply wait in your code for an estimated time so all the bulk requests
should be processed (which assumes the client is up for a long time after
BulkProcessor close), or you can write a BulkProcessor wrapper that is
maintaining a counter on the listener calls of "beforeBulk"/"afterBulk"
listener methods so you know when all responses have been received.
I wrote an improved BulkProcessor which provides a "waitForResponses"
method:
Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:
You must flush the BulkProcessor and wait until your code has received
all responses from outstanding requests. The close() method in
BulkProcessor performs a flush - and recently an explicit flush() is also
implemented -
but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see "no node available" or other things.
Since a "flush" is also executed on a periodical interval, you can either
simply wait in your code for an estimated time so all the bulk requests
should be processed (which assumes the client is up for a long time after
BulkProcessor close), or you can write a BulkProcessor wrapper that is
maintaining a counter on the listener calls of "beforeBulk"/"afterBulk"
listener methods so you know when all responses have been received.
I wrote an improved BulkProcessor which provides a "waitForResponses"
method:
Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
yes, I've tested it with bulk failures - it seems to work well.
Internally, BulkProcessor releases semaphores for bulk failures and other
exceptions that can be thrown by the client, like NoNodeAvailableException,
so it shouldn't ever block forever.
Matt
On Wednesday, 23 April 2014 13:47:20 UTC+1, Jörg Prante wrote:
Oh, thanks for the pointer.
Have you tried the blocking close in case of bulk failures? It looks like
it could hang forever...
Personally I would prefer a separate waitForResponses() method on the
semaphore without synchronization, but that is a matter of taste.
On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:
You must flush the BulkProcessor and wait until your code has received
all responses from outstanding requests. The close() method in
BulkProcessor performs a flush - and recently an explicit flush() is also
implemented -
but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see "no node available" or other things.
Since a "flush" is also executed on a periodical interval, you can
either simply wait in your code for an estimated time so all the bulk
requests should be processed (which assumes the client is up for a long
time after BulkProcessor close), or you can write a BulkProcessor wrapper
that is maintaining a counter on the listener calls of
"beforeBulk"/"afterBulk" listener methods so you know when all responses
have been received.
I wrote an improved BulkProcessor which provides a "waitForResponses"
method:
Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
I see the open issue, but where is the code for this?
On Wednesday, April 23, 2014 10:21:13 AM UTC-4, Matt Preston wrote:
Hi,
yes, I've tested it with bulk failures - it seems to work well.
Internally, BulkProcessor releases semaphores for bulk failures and other
exceptions that can be thrown by the client, like NoNodeAvailableException,
so it shouldn't ever block forever.
Matt
On Wednesday, 23 April 2014 13:47:20 UTC+1, Jörg Prante wrote:
Oh, thanks for the pointer.
Have you tried the blocking close in case of bulk failures? It looks like
it could hang forever...
Personally I would prefer a separate waitForResponses() method on the
semaphore without synchronization, but that is a matter of taste.
On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:
You must flush the BulkProcessor and wait until your code has received
all responses from outstanding requests. The close() method in
BulkProcessor performs a flush - and recently an explicit flush() is also
implemented -
but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see "no node available" or other things.
Since a "flush" is also executed on a periodical interval, you can
either simply wait in your code for an estimated time so all the bulk
requests should be processed (which assumes the client is up for a long
time after BulkProcessor close), or you can write a BulkProcessor wrapper
that is maintaining a counter on the listener calls of
"beforeBulk"/"afterBulk" listener methods so you know when all responses
have been received.
I wrote an improved BulkProcessor which provides a "waitForResponses"
method:
Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.