Can I clarify how admin api works for creating an index?

Elastic Stack Elasticsearch
1

We are testing to see if https://github.com/elasticsearch/elasticsearch/issues/#issue/726
is fixed.
Our test still fails - but differently now. Instead of it hanging, we
get an error.
"Failed to execute phase [query_fetch], total failure; shardFailures
{[na][testindex][0]: No active shards}"

We are creating the index thus:

CreateIndexRequest request = createIndexRequest(indexName);
request.settings("{number_of_shards : 1 }");
client.admin().indices().create(request).actionGet();

Is the index created & ready when "actionGet" method returns?

When we try and search this index immediately after creating it
(theres nothing in the index - we just expect zero results) we get the
above error.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

client.admin()
.cluster()
.health(
clusterHealthRequest()
.waitForYellowStatus() // we only have 1 shard - hence
wait for yellow
.timeout(timeValueSeconds(60))
)
.actionGet();

2

Hi Zohar

Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint

3

There are three different things associated with creating an index. The first is adding it to the cluster metadata, then creating it on each relevant node, and finally, allocating shards to nodes that resulted from the index creation.

The create index API makes sure that the index provided is correct (analyzers are found, mappings are correct), waits for the index itself to be created on the relevant nodes, but does not wait for the shards to be allocated.

The cluster health API, with the wait_for..., allows to wait till shards are allocated. Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

-shay.banon
On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:

Hi Zohar

Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint

4

The create index API makes sure that the index provided is correct
(analyzers are found, mappings are correct), waits for the index itself to
be created on the relevant nodes, but does not wait for the shards to be allocated.

What's the rationale for not waiting for the shards to be allocated?
(just for my education/understanding)

Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

Again, just for my education/understanding, why doesn't the search
wait?

-Nick

On Mar 15, 4:02 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

There are three different things associated with creating an index. The first is adding it to the cluster metadata, then creating it on each relevant node, and finally, allocating shards to nodes that resulted from the index creation.

The create index API makes sure that the index provided is correct (analyzers are found, mappings are correct), waits for the index itself to be created on the relevant nodes, but does not wait for the shards to be allocated.

The cluster health API, with the wait_for..., allows to wait till shards are allocated. Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

-shay.banon

On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:

Hi Zohar

Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint

5

No specific reason except for the complexity involved in implementing it, and the fact that it mainly when doing testing.
On Tuesday, March 15, 2011 at 7:46 PM, Mooky wrote:

The create index API makes sure that the index provided is correct
(analyzers are found, mappings are correct), waits for the index itself to
be created on the relevant nodes, but does not wait for the shards to be allocated.

What's the rationale for not waiting for the shards to be allocated?
(just for my education/understanding)

Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

Again, just for my education/understanding, why doesn't the search
wait?

-Nick

On Mar 15, 4:02 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

There are three different things associated with creating an index. The first is adding it to the cluster metadata, then creating it on each relevant node, and finally, allocating shards to nodes that resulted from the index creation.

The create index API makes sure that the index provided is correct (analyzers are found, mappings are correct), waits for the index itself to be created on the relevant nodes, but does not wait for the shards to be allocated.

The cluster health API, with the wait_for..., allows to wait till shards are allocated. Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

-shay.banon

On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:

Hi Zohar

Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint

6

Also, you might actually want to get search response when not all shards are available.
On Tuesday, March 15, 2011 at 9:25 PM, Shay Banon wrote:

No specific reason except for the complexity involved in implementing it, and the fact that it mainly when doing testing.
On Tuesday, March 15, 2011 at 7:46 PM, Mooky wrote:

The create index API makes sure that the index provided is correct
(analyzers are found, mappings are correct), waits for the index itself to
be created on the relevant nodes, but does not wait for the shards to be allocated.

What's the rationale for not waiting for the shards to be allocated?
(just for my education/understanding)

Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

Again, just for my education/understanding, why doesn't the search
wait?

-Nick

On Mar 15, 4:02 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

There are three different things associated with creating an index. The first is adding it to the cluster metadata, then creating it on each relevant node, and finally, allocating shards to nodes that resulted from the index creation.

The create index API makes sure that the index provided is correct (analyzers are found, mappings are correct), waits for the index itself to be created on the relevant nodes, but does not wait for the shards to be allocated.

The cluster health API, with the wait_for..., allows to wait till shards are allocated. Some APIs wait (till a timeout) if a shard is not allocated yet (like the index API), some don't, like search.

-shay.banon

On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:

Hi Zohar

Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint

7

Ah, yep, true.

On 15 March 2011 19:28, Shay Banon shay.banon@elasticsearch.com wrote:

Also, you might actually want to get search response when not all shards are
available.

8

Fair enough. Thanks.

Yeah, I think in our situation it showed up as a test that kept
failing intermittently.... just a timing thing (initially it was
hanging)

-Nick

On 15 March 2011 19:25, Shay Banon shay.banon@elasticsearch.com wrote:

No specific reason except for the complexity involved in implementing it,
and the fact that it mainly when doing testing.

On Tuesday, March 15, 2011 at 7:46 PM, Mooky wrote:

The create index API makes sure that the index provided is correct
(analyzers are found, mappings are correct), waits for the index itself to
be created on the relevant nodes, but does not wait for the shards to be
allocated.

What's the rationale for not waiting for the shards to be allocated?
(just for my education/understanding)

Some APIs wait (till a timeout) if a shard is not allocated yet (like the
index API), some don't, like search.

Again, just for my education/understanding, why doesn't the search
wait?

-Nick

On Mar 15, 4:02 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

There are three different things associated with creating an index. The
first is adding it to the cluster metadata, then creating it on each
relevant node, and finally, allocating shards to nodes that resulted from
the index creation.

The create index API makes sure that the index provided is correct
(analyzers are found, mappings are correct), waits for the index itself to
be created on the relevant nodes, but does not wait for the shards to be
allocated.

The cluster health API, with the wait_for..., allows to wait till shards are
allocated. Some APIs wait (till a timeout) if a shard is not allocated yet
(like the index API), some don't, like search.

-shay.banon

On Tuesday, March 15, 2011 at 5:32 PM, Clinton Gormley wrote:

Hi Zohar

Is the index created & ready when "actionGet" method returns?

As I understand it (and I stand under correction), a create index call
is async, so it is not ready when actionGet returns.

Out of curiosity, we also added the following code, after creating the
index.
This makes it happy - but not sure if thats just because the call
introduces sufficient delay - or whether its what we need to do.

Not a coincidence. Using a wait_for_status call is the correct thing to
do - exactly what it is designed for.

clint