BulkProcessor close function


(IronMike) #1

bulkProcessor.close();

Does this also close the transportClient? When I follow up the above line
with client.close(), I get no available exception.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #2

No, BulkProcessor does not close the client.

Jörg

On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabdalla80@gmail.com wrote:

bulkProcessor.close();

Does this also close the transportClient? When I follow up the above line
with client.close(), I get no available exception.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFyhW8_8AuztKgZG9j814rD9pgNVw%3DGQUTc-Y%2B9XvBwgQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(IronMike) #3

Ok, so how do I close the client? I have something like this:

main(){

MyIndexer indexer();

for (doc=0; doc <n; doc ++)
indexer.pushDocumentToBulk(doc);

indexer.Shutdown();

}

class MyIndexer(){

//create client

// MyBulkProcesor myBulk;

//PushDocumentToBulk(Doc)

void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here, my
guess bulk is still processing ?
}

}

class MyBulkProcessor(){

BlulkProcessor bulk;

@BeforeBulk ...
@AfterBulk ....

PushDocumentToBulk() { ..push doc ..}

void Close(){ bulk.close(); }
}

On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:

No, BulkProcessor does not close the client.

Jörg

On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 <sabda...@gmail.com<javascript:>

wrote:

bulkProcessor.close();

Does this also close the transportClient? When I follow up the above line
with client.close(), I get no available exception.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/70253f01-fb50-46fe-aa9b-0336d5b70d1c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(IronMike) #4

Ok, so how do I close the client? I have something like this?

main(){

MyIndexer indexer();

for (doc=0; doc <n; doc ++)
indexer.pushDocumentToBulk(doc);

indexer.Shutdown();

}

class MyIndexer(){

//create client

// MyBulkProcesor myBulk;

//PushDocumentToBulk(Doc)

void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here, my
guess bulk is still processing ?
}

}

class MyBulkProcessor(){

BlulkProcessor bulk;

@BeforeBulk ...
@AfterBulk ....

PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}

On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:

No, BulkProcessor does not close the client.

Jörg

On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 <sabda...@gmail.com<javascript:>

wrote:

bulkProcessor.close();

Does this also close the transportClient? When I follow up the above line
with client.close(), I get no available exception.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #5

You must flush the BulkProcessor and wait until your code has received all
responses from outstanding requests. The close() method in BulkProcessor
performs a flush - and recently an explicit flush() is also implemented -

but BulkProcessor does not wait for all responses. If you close your client
prematurely, you will see "no node available" or other things.

Since a "flush" is also executed on a periodical interval, you can either
simply wait in your code for an estimated time so all the bulk requests
should be processed (which assumes the client is up for a long time after
BulkProcessor close), or you can write a BulkProcessor wrapper that is
maintaining a counter on the listener calls of "beforeBulk"/"afterBulk"
listener methods so you know when all responses have been received.

I wrote an improved BulkProcessor which provides a "waitForResponses"
method:

https://github.com/jprante/elasticsearch-support/blob/master/src/main/java/org/xbib/elasticsearch/action/ingest/IngestProcessor.java

Jörg

On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 sabdalla80@gmail.com wrote:

Ok, so how do I close the client? I have something like this?

main(){

MyIndexer indexer();

for (doc=0; doc <n; doc ++)
indexer.pushDocumentToBulk(doc);

indexer.Shutdown();

}

class MyIndexer(){

//create client

// MyBulkProcesor myBulk;

//PushDocumentToBulk(Doc)

void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here, my
guess bulk is still processing ?
}

}

class MyBulkProcessor(){

BlulkProcessor bulk;

@BeforeBulk ...
@AfterBulk ....

PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}

On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:

No, BulkProcessor does not close the client.

Jörg

On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabda...@gmail.com wrote:

bulkProcessor.close();

Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFXCV23bzyz%2BQ3mgS_LM86K4fCJ7WGY1YKL%3DbWVJ_PEyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Matt Preston) #6

There is an open issue to add a blocking close method to BulkProcessor

Matt

On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:

You must flush the BulkProcessor and wait until your code has received all
responses from outstanding requests. The close() method in BulkProcessor
performs a flush - and recently an explicit flush() is also implemented -

https://github.com/elasticsearch/elasticsearch/issues/5570

but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see "no node available" or other things.

Since a "flush" is also executed on a periodical interval, you can either
simply wait in your code for an estimated time so all the bulk requests
should be processed (which assumes the client is up for a long time after
BulkProcessor close), or you can write a BulkProcessor wrapper that is
maintaining a counter on the listener calls of "beforeBulk"/"afterBulk"
listener methods so you know when all responses have been received.

I wrote an improved BulkProcessor which provides a "waitForResponses"
method:

https://github.com/jprante/elasticsearch-support/blob/master/src/main/java/org/xbib/elasticsearch/action/ingest/IngestProcessor.java

Jörg

On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 <sabda...@gmail.com<javascript:>

wrote:

Ok, so how do I close the client? I have something like this?

main(){

MyIndexer indexer();

for (doc=0; doc <n; doc ++)
indexer.pushDocumentToBulk(doc);

indexer.Shutdown();

}

class MyIndexer(){

//create client

// MyBulkProcesor myBulk;

//PushDocumentToBulk(Doc)

void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here, my
guess bulk is still processing ?
}

}

class MyBulkProcessor(){

BlulkProcessor bulk;

@BeforeBulk ...
@AfterBulk ....

PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}

On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:

No, BulkProcessor does not close the client.

Jörg

On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabda...@gmail.comwrote:

bulkProcessor.close();

Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #7

Oh, thanks for the pointer.

Have you tried the blocking close in case of bulk failures? It looks like
it could hang forever...

Personally I would prefer a separate waitForResponses() method on the
semaphore without synchronization, but that is a matter of taste.

Jörg

On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston <
matthew.preston@thomsonreuters.com> wrote:

There is an open issue to add a blocking close method to BulkProcessor

https://github.com/elasticsearch/elasticsearch/pull/4180

Matt

On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:

You must flush the BulkProcessor and wait until your code has received
all responses from outstanding requests. The close() method in
BulkProcessor performs a flush - and recently an explicit flush() is also
implemented -

https://github.com/elasticsearch/elasticsearch/issues/5570

but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see "no node available" or other things.

Since a "flush" is also executed on a periodical interval, you can either
simply wait in your code for an estimated time so all the bulk requests
should be processed (which assumes the client is up for a long time after
BulkProcessor close), or you can write a BulkProcessor wrapper that is
maintaining a counter on the listener calls of "beforeBulk"/"afterBulk"
listener methods so you know when all responses have been received.

I wrote an improved BulkProcessor which provides a "waitForResponses"
method:

https://github.com/jprante/elasticsearch-support/blob/
master/src/main/java/org/xbib/elasticsearch/action/ingest/
IngestProcessor.java

Jörg

On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 sabda...@gmail.com wrote:

Ok, so how do I close the client? I have something like this?

main(){

MyIndexer indexer();

for (doc=0; doc <n; doc ++)
indexer.pushDocumentToBulk(doc);

indexer.Shutdown();

}

class MyIndexer(){

//create client

// MyBulkProcesor myBulk;

//PushDocumentToBulk(Doc)

void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here,
my guess bulk is still processing ?
}

}

class MyBulkProcessor(){

BlulkProcessor bulk;

@BeforeBulk ...
@AfterBulk ....

PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}

On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:

No, BulkProcessor does not close the client.

Jörg

On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabda...@gmail.comwrote:

bulkProcessor.close();

Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvQQjwBRuSU7Ke3iUp6%3D%2BQBX-LznqKrYf3yF59Rm3Dvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Matt Preston) #8

Hi,

yes, I've tested it with bulk failures - it seems to work well.
Internally, BulkProcessor releases semaphores for bulk failures and other
exceptions that can be thrown by the client, like NoNodeAvailableException,
so it shouldn't ever block forever.

Matt

On Wednesday, 23 April 2014 13:47:20 UTC+1, Jörg Prante wrote:

Oh, thanks for the pointer.

Have you tried the blocking close in case of bulk failures? It looks like
it could hang forever...

Personally I would prefer a separate waitForResponses() method on the
semaphore without synchronization, but that is a matter of taste.

Jörg

On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston <
matthew...@thomsonreuters.com <javascript:>> wrote:

There is an open issue to add a blocking close method to BulkProcessor

https://github.com/elasticsearch/elasticsearch/pull/4180

Matt

On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:

You must flush the BulkProcessor and wait until your code has received
all responses from outstanding requests. The close() method in
BulkProcessor performs a flush - and recently an explicit flush() is also
implemented -

https://github.com/elasticsearch/elasticsearch/issues/5570

but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see "no node available" or other things.

Since a "flush" is also executed on a periodical interval, you can
either simply wait in your code for an estimated time so all the bulk
requests should be processed (which assumes the client is up for a long
time after BulkProcessor close), or you can write a BulkProcessor wrapper
that is maintaining a counter on the listener calls of
"beforeBulk"/"afterBulk" listener methods so you know when all responses
have been received.

I wrote an improved BulkProcessor which provides a "waitForResponses"
method:

https://github.com/jprante/elasticsearch-support/blob/
master/src/main/java/org/xbib/elasticsearch/action/ingest/
IngestProcessor.java

Jörg

On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 sabda...@gmail.comwrote:

Ok, so how do I close the client? I have something like this?

main(){

MyIndexer indexer();

for (doc=0; doc <n; doc ++)
indexer.pushDocumentToBulk(doc);

indexer.Shutdown();

}

class MyIndexer(){

//create client

// MyBulkProcesor myBulk;

//PushDocumentToBulk(Doc)

void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here,
my guess bulk is still processing ?
}

}

class MyBulkProcessor(){

BlulkProcessor bulk;

@BeforeBulk ...
@AfterBulk ....

PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}

On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:

No, BulkProcessor does not close the client.

Jörg

On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabda...@gmail.comwrote:

bulkProcessor.close();

Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b637cef4-35a3-4d3a-92d9-0334189cd112%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(IronMike) #9

Matt,

I see the open issue, but where is the code for this?

On Wednesday, April 23, 2014 10:21:13 AM UTC-4, Matt Preston wrote:

Hi,

yes, I've tested it with bulk failures - it seems to work well.
Internally, BulkProcessor releases semaphores for bulk failures and other
exceptions that can be thrown by the client, like NoNodeAvailableException,
so it shouldn't ever block forever.

Matt

On Wednesday, 23 April 2014 13:47:20 UTC+1, Jörg Prante wrote:

Oh, thanks for the pointer.

Have you tried the blocking close in case of bulk failures? It looks like
it could hang forever...

Personally I would prefer a separate waitForResponses() method on the
semaphore without synchronization, but that is a matter of taste.

Jörg

On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston <
matthew...@thomsonreuters.com> wrote:

There is an open issue to add a blocking close method to BulkProcessor

https://github.com/elasticsearch/elasticsearch/pull/4180

Matt

On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:

You must flush the BulkProcessor and wait until your code has received
all responses from outstanding requests. The close() method in
BulkProcessor performs a flush - and recently an explicit flush() is also
implemented -

https://github.com/elasticsearch/elasticsearch/issues/5570

but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see "no node available" or other things.

Since a "flush" is also executed on a periodical interval, you can
either simply wait in your code for an estimated time so all the bulk
requests should be processed (which assumes the client is up for a long
time after BulkProcessor close), or you can write a BulkProcessor wrapper
that is maintaining a counter on the listener calls of
"beforeBulk"/"afterBulk" listener methods so you know when all responses
have been received.

I wrote an improved BulkProcessor which provides a "waitForResponses"
method:

https://github.com/jprante/elasticsearch-support/blob/
master/src/main/java/org/xbib/elasticsearch/action/ingest/
IngestProcessor.java

Jörg

On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 sabda...@gmail.comwrote:

Ok, so how do I close the client? I have something like this?

main(){

MyIndexer indexer();

for (doc=0; doc <n; doc ++)
indexer.pushDocumentToBulk(doc);

indexer.Shutdown();

}

class MyIndexer(){

//create client

// MyBulkProcesor myBulk;

//PushDocumentToBulk(Doc)

void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here,
my guess bulk is still processing ?
}

}

class MyBulkProcessor(){

BlulkProcessor bulk;

@BeforeBulk ...
@AfterBulk ....

PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}

On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:

No, BulkProcessor does not close the client.

Jörg

On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabda...@gmail.comwrote:

bulkProcessor.close();

Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a923766f-1f7
0-4e78-8b0f-f443e16a2cd7%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c331a55c-69e4-4538-b63f-aee8fc3f9375%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #10