Simultaneous indexing and searching in 2 threads gets "Failed to execute phase" exception

Elastic Stack Elasticsearch
1

I have a situation where, using the Java API, I initiate a bunch of
indexing operations, but throw away the Future object (I don't need the
return status). This is so I can do a lot of indexing reasonably
asynchronously, so I don't have to hold up the GUI that triggers these
calls.

However, if I fire off these indexing operations, and immediately after do
a search operation on the same index, I get the following exception:

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException:
Failed to execute phase [init_scan], all shards failed
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233)
[elasticsearch-1.4.1.jar:]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179)
[elasticsearch-1.4.1.jar:]
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565)
[elasticsearch-1.4.1.jar:]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_60]
... 1 more

... however, if I wait awhile after indexing before searching, the search
succeeds, no errors.

Does anyone know what is going on there? Shouldn't initiating an index
operation asynchronously not affect search operations made in a different
client?

This is happening only when I use 2 separate threads (and two separate
TransportClients) for each operation... thus:

  1. Thread 1: Fork Thread 2

  2. Thread 2: Create new TransportClient; Index list of items, not waiting
    for Future objects (i.e. not calling actionGet())

  3. Thread 1: While Thread 2 is running, create new TransportClient and do
    search

  4. Get exception above

... compared to:

  1. Thread 1: Fork Thread 2

  2. Thread 2: Create new TransportClient; Index list of items, not waiting
    for Future objects (i.e. not calling actionGet())

  3. Wait a bit

  4. Thread 1: (Thread 2 has presumably finished by now); create new
    TransportClient and do search

  5. Don't get exception above

... also, if, instead, I do it one thread:

  1. Create new TransportClient; Index list of items, not waiting for Future
    objects (i.e. not calling actionGet())
  2. Close TransportClient
  3. Create new TransportClient; do search

... it works fine. So it seems to have something to do with simultaneous
actions on 2 different TransportClients to the same index. It might also
have something to do with internal client state, even though I'm using 2
separate client objects.

Does this sound unusual to anyone?

  • Tim

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Some ideas:

You can/should share the same client within all threads. So only one client for the full JVM.
You should create first the index and wait for the index to be created, using actionGet(). It's a quick operation. Then run your code as you wrote.

My 2 cents.

David

Le 22 janv. 2015 à 06:12, TimOnGmail timbessie@gmail.com a écrit :

I have a situation where, using the Java API, I initiate a bunch of indexing operations, but throw away the Future object (I don't need the return status). This is so I can do a lot of indexing reasonably asynchronously, so I don't have to hold up the GUI that triggers these calls.

However, if I fire off these indexing operations, and immediately after do a search operation on the same index, I get the following exception:

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [init_scan], all shards failed
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233) [elasticsearch-1.4.1.jar:]
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179) [elasticsearch-1.4.1.jar:]
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565) [elasticsearch-1.4.1.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
... 1 more

... however, if I wait awhile after indexing before searching, the search succeeds, no errors.

Does anyone know what is going on there? Shouldn't initiating an index operation asynchronously not affect search operations made in a different client?

This is happening only when I use 2 separate threads (and two separate TransportClients) for each operation... thus:

  1. Thread 1: Fork Thread 2
  2. Thread 2: Create new TransportClient; Index list of items, not waiting for Future objects (i.e. not calling actionGet())
  3. Thread 1: While Thread 2 is running, create new TransportClient and do search
  4. Get exception above

... compared to:

  1. Thread 1: Fork Thread 2
  2. Thread 2: Create new TransportClient; Index list of items, not waiting for Future objects (i.e. not calling actionGet())
  3. Wait a bit
  4. Thread 1: (Thread 2 has presumably finished by now); create new TransportClient and do search
  5. Don't get exception above

... also, if, instead, I do it one thread:

  1. Create new TransportClient; Index list of items, not waiting for Future objects (i.e. not calling actionGet())
  2. Close TransportClient
  3. Create new TransportClient; do search

... it works fine. So it seems to have something to do with simultaneous actions on 2 different TransportClients to the same index. It might also have something to do with internal client state, even though I'm using 2 separate client objects.

Does this sound unusual to anyone?

  • Tim

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2B6FE107-606F-4314-8CC8-D5708ECCFD6F%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

Thanks for your suggestions!

I'm generally created a fresh client for each index/search request. So
that's not correct? I had thought it was better to do it that way.

Any problems with using separate clients in the same VM, different threads,
that you know of?

The indexes are already created, incidentally; the calls I'm making are in
adding items to the index or in searching the index only.

  • Tim

On Wednesday, January 21, 2015 at 10:57:52 PM UTC-8, David Pilato wrote:

Some ideas:

You can/should share the same client within all threads. So only one
client for the full JVM.
You should create first the index and wait for the index to be created,
using actionGet(). It's a quick operation. Then run your code as you wrote.

My 2 cents.

David

Le 22 janv. 2015 à 06:12, TimOnGmail <timb...@gmail.com <javascript:>> a
écrit :

I have a situation where, using the Java API, I initiate a bunch of
indexing operations, but throw away the Future object (I don't need the
return status). This is so I can do a lot of indexing reasonably
asynchronously, so I don't have to hold up the GUI that triggers these
calls.

However, if I fire off these indexing operations, and immediately after do
a search operation on the same index, I get the following exception:

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException:
Failed to execute phase [init_scan], all shards failed
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233)
[elasticsearch-1.4.1.jar:]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179)
[elasticsearch-1.4.1.jar:]
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565)
[elasticsearch-1.4.1.jar:]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_60]
... 1 more

... however, if I wait awhile after indexing before searching, the search
succeeds, no errors.

Does anyone know what is going on there? Shouldn't initiating an index
operation asynchronously not affect search operations made in a different
client?

This is happening only when I use 2 separate threads (and two separate
TransportClients) for each operation... thus:

  1. Thread 1: Fork Thread 2

  2. Thread 2: Create new TransportClient; Index list of items, not waiting
    for Future objects (i.e. not calling actionGet())

  3. Thread 1: While Thread 2 is running, create new TransportClient and do
    search

  4. Get exception above

... compared to:

  1. Thread 1: Fork Thread 2

  2. Thread 2: Create new TransportClient; Index list of items, not waiting
    for Future objects (i.e. not calling actionGet())

  3. Wait a bit

  4. Thread 1: (Thread 2 has presumably finished by now); create new
    TransportClient and do search

  5. Don't get exception above

... also, if, instead, I do it one thread:

  1. Create new TransportClient; Index list of items, not waiting for Future
    objects (i.e. not calling actionGet())
  2. Close TransportClient
  3. Create new TransportClient; do search

... it works fine. So it seems to have something to do with simultaneous
actions on 2 different TransportClients to the same index. It might also
have something to do with internal client state, even though I'm using 2
separate client objects.

Does this sound unusual to anyone?

  • Tim

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b81b3d52-a3c0-4222-9009-117ea6f28c93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Answers inlined

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 janv. 2015 à 20:19, TimOnGmail timbessie@gmail.com a écrit :

Thanks for your suggestions!

I'm generally created a fresh client for each index/search request. So that's not correct? I had thought it was better to do it that way.

No. You should create a singleton.

Any problems with using separate clients in the same VM, different threads, that you know of?

You don't need them so that's a waste of resources IMHO.

The indexes are already created, incidentally; the calls I'm making are in adding items to the index or in searching the index only.

That's super strange that shards are failing if the index has been created with success.

  • Tim

On Wednesday, January 21, 2015 at 10:57:52 PM UTC-8, David Pilato wrote:
Some ideas:

You can/should share the same client within all threads. So only one client for the full JVM.
You should create first the index and wait for the index to be created, using actionGet(). It's a quick operation. Then run your code as you wrote.

My 2 cents.

David

Le 22 janv. 2015 à 06:12, TimOnGmail timb...@gmail.com a écrit :

I have a situation where, using the Java API, I initiate a bunch of indexing operations, but throw away the Future object (I don't need the return status). This is so I can do a lot of indexing reasonably asynchronously, so I don't have to hold up the GUI that triggers these calls.

However, if I fire off these indexing operations, and immediately after do a search operation on the same index, I get the following exception:

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [init_scan], all shards failed
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233) [elasticsearch-1.4.1.jar:]
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179) [elasticsearch-1.4.1.jar:]
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565) [elasticsearch-1.4.1.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
... 1 more

... however, if I wait awhile after indexing before searching, the search succeeds, no errors.

Does anyone know what is going on there? Shouldn't initiating an index operation asynchronously not affect search operations made in a different client?

This is happening only when I use 2 separate threads (and two separate TransportClients) for each operation... thus:

  1. Thread 1: Fork Thread 2
  2. Thread 2: Create new TransportClient; Index list of items, not waiting for Future objects (i.e. not calling actionGet())
  3. Thread 1: While Thread 2 is running, create new TransportClient and do search
  4. Get exception above

... compared to:

  1. Thread 1: Fork Thread 2
  2. Thread 2: Create new TransportClient; Index list of items, not waiting for Future objects (i.e. not calling actionGet())
  3. Wait a bit
  4. Thread 1: (Thread 2 has presumably finished by now); create new TransportClient and do search
  5. Don't get exception above

... also, if, instead, I do it one thread:

  1. Create new TransportClient; Index list of items, not waiting for Future objects (i.e. not calling actionGet())
  2. Close TransportClient
  3. Create new TransportClient; do search

... it works fine. So it seems to have something to do with simultaneous actions on 2 different TransportClients to the same index. It might also have something to do with internal client state, even though I'm using 2 separate client objects.

Does this sound unusual to anyone?

  • Tim

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b81b3d52-a3c0-4222-9009-117ea6f28c93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/38937576-391B-4341-B861-338E389B5FFD%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

5

I changed the code to use a Singleton. Even so, when I made the indexing
and searching happen in 2 different threads (without waiting for responses

  • just ignoring the returned future), it failed similarly - even if I
    waited for awhile before issuing the search.

If I did the same thing, but didn't do it in 2 threads, it succeeded.

So I am wondering if there is some issue when accessing the Singleton
client in different threads, some non-threadsafe issue at work.

David, are you a dev on the Elasticsearch team? If not, I hope one of them
chimes in here.

  • Tim

On Thursday, January 22, 2015 at 2:30:07 PM UTC-8, David Pilato wrote:

Answers inlined

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 janv. 2015 à 20:19, TimOnGmail <timb...@gmail.com <javascript:>> a
écrit :

Thanks for your suggestions!

I'm generally created a fresh client for each index/search request. So
that's not correct? I had thought it was better to do it that way.

No. You should create a singleton.

Any problems with using separate clients in the same VM, different
threads, that you know of?

You don't need them so that's a waste of resources IMHO.

The indexes are already created, incidentally; the calls I'm making are in
adding items to the index or in searching the index only.

That's super strange that shards are failing if the index has been created
with success.

  • Tim

On Wednesday, January 21, 2015 at 10:57:52 PM UTC-8, David Pilato wrote:

Some ideas:

You can/should share the same client within all threads. So only one
client for the full JVM.
You should create first the index and wait for the index to be created,
using actionGet(). It's a quick operation. Then run your code as you wrote.

My 2 cents.

David

Le 22 janv. 2015 à 06:12, TimOnGmail timb...@gmail.com a écrit :

I have a situation where, using the Java API, I initiate a bunch of
indexing operations, but throw away the Future object (I don't need the
return status). This is so I can do a lot of indexing reasonably
asynchronously, so I don't have to hold up the GUI that triggers these
calls.

However, if I fire off these indexing operations, and immediately after
do a search operation on the same index, I get the following exception:

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException:
Failed to execute phase [init_scan], all shards failed
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233)
[elasticsearch-1.4.1.jar:]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179)
[elasticsearch-1.4.1.jar:]
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565)
[elasticsearch-1.4.1.jar:]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_60]
... 1 more

... however, if I wait awhile after indexing before searching, the search
succeeds, no errors.

Does anyone know what is going on there? Shouldn't initiating an index
operation asynchronously not affect search operations made in a different
client?

This is happening only when I use 2 separate threads (and two separate
TransportClients) for each operation... thus:

  1. Thread 1: Fork Thread 2

  2. Thread 2: Create new TransportClient; Index list of items, not waiting
    for Future objects (i.e. not calling actionGet())

  3. Thread 1: While Thread 2 is running, create new TransportClient and do
    search

  4. Get exception above

... compared to:

  1. Thread 1: Fork Thread 2

  2. Thread 2: Create new TransportClient; Index list of items, not waiting
    for Future objects (i.e. not calling actionGet())

  3. Wait a bit

  4. Thread 1: (Thread 2 has presumably finished by now); create new
    TransportClient and do search

  5. Don't get exception above

... also, if, instead, I do it one thread:

  1. Create new TransportClient; Index list of items, not waiting for
    Future objects (i.e. not calling actionGet())
  2. Close TransportClient
  3. Create new TransportClient; do search

... it works fine. So it seems to have something to do with simultaneous
actions on 2 different TransportClients to the same index. It might also
have something to do with internal client state, even though I'm using 2
separate client objects.

Does this sound unusual to anyone?

  • Tim

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b81b3d52-a3c0-4222-9009-117ea6f28c93%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b81b3d52-a3c0-4222-9009-117ea6f28c93%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b269a81f-dda7-4984-95e0-1e0435da67af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Yes I am.

Client is thread safe. Any chance you could share on github a small project which reproduce this error?

David

Le 23 janv. 2015 à 01:18, TimOnGmail timbessie@gmail.com a écrit :

I changed the code to use a Singleton. Even so, when I made the indexing and searching happen in 2 different threads (without waiting for responses - just ignoring the returned future), it failed similarly - even if I waited for awhile before issuing the search.

If I did the same thing, but didn't do it in 2 threads, it succeeded.

So I am wondering if there is some issue when accessing the Singleton client in different threads, some non-threadsafe issue at work.

David, are you a dev on the Elasticsearch team? If not, I hope one of them chimes in here.

  • Tim

On Thursday, January 22, 2015 at 2:30:07 PM UTC-8, David Pilato wrote:
Answers inlined

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 janv. 2015 à 20:19, TimOnGmail timb...@gmail.com a écrit :

Thanks for your suggestions!

I'm generally created a fresh client for each index/search request. So that's not correct? I had thought it was better to do it that way.

No. You should create a singleton.

Any problems with using separate clients in the same VM, different threads, that you know of?

You don't need them so that's a waste of resources IMHO.

The indexes are already created, incidentally; the calls I'm making are in adding items to the index or in searching the index only.

That's super strange that shards are failing if the index has been created with success.

  • Tim

On Wednesday, January 21, 2015 at 10:57:52 PM UTC-8, David Pilato wrote:
Some ideas:

You can/should share the same client within all threads. So only one client for the full JVM.
You should create first the index and wait for the index to be created, using actionGet(). It's a quick operation. Then run your code as you wrote.

My 2 cents.

David

Le 22 janv. 2015 à 06:12, TimOnGmail timb...@gmail.com a écrit :

I have a situation where, using the Java API, I initiate a bunch of indexing operations, but throw away the Future object (I don't need the return status). This is so I can do a lot of indexing reasonably asynchronously, so I don't have to hold up the GUI that triggers these calls.

However, if I fire off these indexing operations, and immediately after do a search operation on the same index, I get the following exception:

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [init_scan], all shards failed
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233) [elasticsearch-1.4.1.jar:]
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179) [elasticsearch-1.4.1.jar:]
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565) [elasticsearch-1.4.1.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
... 1 more

... however, if I wait awhile after indexing before searching, the search succeeds, no errors.

Does anyone know what is going on there? Shouldn't initiating an index operation asynchronously not affect search operations made in a different client?

This is happening only when I use 2 separate threads (and two separate TransportClients) for each operation... thus:

  1. Thread 1: Fork Thread 2
  2. Thread 2: Create new TransportClient; Index list of items, not waiting for Future objects (i.e. not calling actionGet())
  3. Thread 1: While Thread 2 is running, create new TransportClient and do search
  4. Get exception above

... compared to:

  1. Thread 1: Fork Thread 2
  2. Thread 2: Create new TransportClient; Index list of items, not waiting for Future objects (i.e. not calling actionGet())
  3. Wait a bit
  4. Thread 1: (Thread 2 has presumably finished by now); create new TransportClient and do search
  5. Don't get exception above

... also, if, instead, I do it one thread:

  1. Create new TransportClient; Index list of items, not waiting for Future objects (i.e. not calling actionGet())
  2. Close TransportClient
  3. Create new TransportClient; do search

... it works fine. So it seems to have something to do with simultaneous actions on 2 different TransportClients to the same index. It might also have something to do with internal client state, even though I'm using 2 separate client objects.

Does this sound unusual to anyone?

  • Tim

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b81b3d52-a3c0-4222-9009-117ea6f28c93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b269a81f-dda7-4984-95e0-1e0435da67af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/C324892D-BDB5-46BE-A81F-D6437DBE4FDD%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

7

I could do that - I'll have to see if I can trim it down to a reasonable
size without all the unrelated code in it.

Is it obvious on GitHub where people can submit error samples? I haven't
done that before.

  • Tim

On Friday, January 23, 2015 at 1:27:05 AM UTC-8, David Pilato wrote:

Yes I am.

Client is thread safe. Any chance you could share on github a small
project which reproduce this error?

David

Le 23 janv. 2015 à 01:18, TimOnGmail <timb...@gmail.com <javascript:>> a
écrit :

I changed the code to use a Singleton. Even so, when I made the indexing
and searching happen in 2 different threads (without waiting for responses

  • just ignoring the returned future), it failed similarly - even if I
    waited for awhile before issuing the search.

If I did the same thing, but didn't do it in 2 threads, it succeeded.

So I am wondering if there is some issue when accessing the Singleton
client in different threads, some non-threadsafe issue at work.

David, are you a dev on the Elasticsearch team? If not, I hope one of
them chimes in here.

  • Tim

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c405d2af-cc0f-4803-8197-7791b0133845%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

It always easier to share a project for which we only have to do:

git clone
mvn install

But, you could also share on gist.github.com a pom.xml and a test class.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 26 janv. 2015 à 19:27, TimOnGmail timbessie@gmail.com a écrit :

I could do that - I'll have to see if I can trim it down to a reasonable size without all the unrelated code in it.

Is it obvious on GitHub where people can submit error samples? I haven't done that before.

  • Tim

On Friday, January 23, 2015 at 1:27:05 AM UTC-8, David Pilato wrote:
Yes I am.

Client is thread safe. Any chance you could share on github a small project which reproduce this error?

David

Le 23 janv. 2015 à 01:18, TimOnGmail timb...@gmail.com a écrit :

I changed the code to use a Singleton. Even so, when I made the indexing and searching happen in 2 different threads (without waiting for responses - just ignoring the returned future), it failed similarly - even if I waited for awhile before issuing the search.

If I did the same thing, but didn't do it in 2 threads, it succeeded.

So I am wondering if there is some issue when accessing the Singleton client in different threads, some non-threadsafe issue at work.

David, are you a dev on the Elasticsearch team? If not, I hope one of them chimes in here.

  • Tim

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c405d2af-cc0f-4803-8197-7791b0133845%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/DED74DF5-CC97-49AB-9F70-A220C33E8A1F%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

9

Without seeing the code, I guess your ES cluster is a bit slow in starting
up new shards, which is pretty normal. There is a small time span after
index creation and getting ready for accepting docs. You should wait for
the created index to get initialized. Usually this is not required, because
actionGet() synchronizes this for you, or the index was already created in
an explicit call.

Jörg

On Fri, Jan 23, 2015 at 1:18 AM, TimOnGmail timbessie@gmail.com wrote:

I changed the code to use a Singleton. Even so, when I made the indexing
and searching happen in 2 different threads (without waiting for responses

  • just ignoring the returned future), it failed similarly - even if I
    waited for awhile before issuing the search.

If I did the same thing, but didn't do it in 2 threads, it succeeded.

So I am wondering if there is some issue when accessing the Singleton
client in different threads, some non-threadsafe issue at work.

David, are you a dev on the Elasticsearch team? If not, I hope one of
them chimes in here.

  • Tim

On Thursday, January 22, 2015 at 2:30:07 PM UTC-8, David Pilato wrote:

Answers inlined

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 janv. 2015 à 20:19, TimOnGmail timb...@gmail.com a écrit :

Thanks for your suggestions!

I'm generally created a fresh client for each index/search request. So
that's not correct? I had thought it was better to do it that way.

No. You should create a singleton.

Any problems with using separate clients in the same VM, different
threads, that you know of?

You don't need them so that's a waste of resources IMHO.

The indexes are already created, incidentally; the calls I'm making are
in adding items to the index or in searching the index only.

That's super strange that shards are failing if the index has been
created with success.

  • Tim

On Wednesday, January 21, 2015 at 10:57:52 PM UTC-8, David Pilato wrote:

Some ideas:

You can/should share the same client within all threads. So only one
client for the full JVM.
You should create first the index and wait for the index to be created,
using actionGet(). It's a quick operation. Then run your code as you wrote.

My 2 cents.

David

Le 22 janv. 2015 à 06:12, TimOnGmail timb...@gmail.com a écrit :

I have a situation where, using the Java API, I initiate a bunch of
indexing operations, but throw away the Future object (I don't need the
return status). This is so I can do a lot of indexing reasonably
asynchronously, so I don't have to hold up the GUI that triggers these
calls.

However, if I fire off these indexing operations, and immediately after
do a search operation on the same index, I get the following exception:

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException:
Failed to execute phase [init_scan], all shards failed
at org.elasticsearch.action.search.type.TransportSearchTypeAction$
BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233)
[elasticsearch-1.4.1.jar:]
at org.elasticsearch.action.search.type.TransportSearchTypeAction$
BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179)
[elasticsearch-1.4.1.jar:]
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:565) [elasticsearch-1.4.1.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_60]
... 1 more

... however, if I wait awhile after indexing before searching, the
search succeeds, no errors.

Does anyone know what is going on there? Shouldn't initiating an index
operation asynchronously not affect search operations made in a different
client?

This is happening only when I use 2 separate threads (and two separate
TransportClients) for each operation... thus:

  1. Thread 1: Fork Thread 2

  2. Thread 2: Create new TransportClient; Index list of items, not
    waiting for Future objects (i.e. not calling actionGet())

  3. Thread 1: While Thread 2 is running, create new TransportClient and
    do search

  4. Get exception above

... compared to:

  1. Thread 1: Fork Thread 2

  2. Thread 2: Create new TransportClient; Index list of items, not
    waiting for Future objects (i.e. not calling actionGet())

  3. Wait a bit

  4. Thread 1: (Thread 2 has presumably finished by now); create new
    TransportClient and do search

  5. Don't get exception above

... also, if, instead, I do it one thread:

  1. Create new TransportClient; Index list of items, not waiting for
    Future objects (i.e. not calling actionGet())
  2. Close TransportClient
  3. Create new TransportClient; do search

... it works fine. So it seems to have something to do with
simultaneous actions on 2 different TransportClients to the same index. It
might also have something to do with internal client state, even though I'm
using 2 separate client objects.

Does this sound unusual to anyone?

  • Tim

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b1992277-4e72-4ba9-9cdd-f00a6917e55a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/b81b3d52-a3c0-4222-9009-117ea6f28c93%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b81b3d52-a3c0-4222-9009-117ea6f28c93%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b269a81f-dda7-4984-95e0-1e0435da67af%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b269a81f-dda7-4984-95e0-1e0435da67af%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEODGR9ix8-Ni7_DxGhynVLW7U7ouRwM4eTqq98Vc539Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.