Corrupt index creation when elasticsearch is killed just after index is created


(tarang dawer) #1

Hi all
I am facing an issue of corrupt index creation, whenever the es node is
killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.

Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #2

Hey,

can you reliably recreate this? And try to create a gist? Preferrably, when
using elasticsearch 0.90.9. When you create an index, you usually create 5
shards, so, how can 3/4 shards be corrupt? Did you change anything and do
not use the defaults (are you changing the defaults somewhere else as
well)? It would be great if you could provide a reproducible example using
a gist, like mentioned in http://www.elasticsearch.org/help

--Alex

On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer tarang.dawer@gmail.comwrote:

Hi all
I am facing an issue of corrupt index creation, whenever the es node is
killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.

Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(tarang dawer) #3

i have reliably recreated this many times, happens while creating index on
a single node, (default 5 shards). i have set "action.auto_create_index:
false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true"
so i am creating indices via java API, . i kill(Kill -9 ) the elasticsearch
immediately after the index is created. when i restart the elasticsearch,
out of the 5 primary shards, it shows 3/4 shards in a corrupt state, with
"503" status code.

On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen alr@spinscale.de wrote:

Hey,

can you reliably recreate this? And try to create a gist? Preferrably,
when using elasticsearch 0.90.9. When you create an index, you usually
create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
and do not use the defaults (are you changing the defaults somewhere else
as well)? It would be great if you could provide a reproducible example
using a gist, like mentioned in http://www.elasticsearch.org/help

--Alex

On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer tarang.dawer@gmail.comwrote:

Hi all
I am facing an issue of corrupt index creation, whenever the es node is
killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.

Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUrAfA6Qe_NDEQ9SwRWmOK%2BkSLY%3Dq%3D9syENAZ_yx3B5yg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(tarang dawer) #4

tried this on 90.9 also . This time after the restart when i get the index
status as :-

In the elasticsearch head plugin, it shows the cluster is in red state,
with no shards being allocated to the node even after the restart of
elasticsearch.

http://localhost:9200/indexName/_status

{"ok":true,"_shards":{"total":10,"successful":0,"failed":0},"indices":{}}

In the elasticsearch head plugin, it shows the cluster is in red
state, with no shards being allocated to the node even
after the restart of elasticsearch.

Squirrel Girlcluster health: red (1, 0)

On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer tarang.dawer@gmail.comwrote:

i have reliably recreated this many times, happens while creating index on
a single node, (default 5 shards). i have set "action.auto_create_index:
false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true"
so i am creating indices via java API, . i kill(Kill -9 ) the elasticsearch
immediately after the index is created. when i restart the elasticsearch,
out of the 5 primary shards, it shows 3/4 shards in a corrupt state, with
"503" status code.

On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey,

can you reliably recreate this? And try to create a gist? Preferrably,
when using elasticsearch 0.90.9. When you create an index, you usually
create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
and do not use the defaults (are you changing the defaults somewhere else
as well)? It would be great if you could provide a reproducible example
using a gist, like mentioned in http://www.elasticsearch.org/help

--Alex

On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer tarang.dawer@gmail.comwrote:

Hi all
I am facing an issue of corrupt index creation, whenever the es node is
killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.

Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUZgt6nFpGgZa3arTAPJXt2ett6tqo1ytG5-XkrK_2_hQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(tarang dawer) #5

Tried one again, this time, the status showed 3 successful shards, but
however the indexing got stuck, and after a while an exception was thrown

Caused by: org.elasticsearch.action.UnavailableShardsException:
[indexName][4] [2] shardIt, [0] active : Timeout waiting for [1m], request:
index {[indexName][indexName][ID], source["Record":"Details"]}
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)
... 3 more

and the cluster state is still in red, with elasticsearch not able to
recover the shards after the restart

Conquer Lordcluster health: red (1, 3)

On Thu, Dec 26, 2013 at 10:32 AM, Tarang Dawer tarang.dawer@gmail.comwrote:

tried this on 90.9 also . This time after the restart when i get the index
status as :-

In the elasticsearch head plugin, it shows the cluster is in red state,
with no shards being allocated to the node even after the restart of
elasticsearch.

http://localhost:9200/indexName/_status

{"ok":true,"_shards":{"total":10,"successful":0,"failed":0},"indices":{}}

In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even
after the restart of elasticsearch.

Squirrel Girlcluster health: red (1, 0)

On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer tarang.dawer@gmail.comwrote:

i have reliably recreated this many times, happens while creating index
on a single node, (default 5 shards). i have set
"action.auto_create_index: false" , "discovery.zen.ping.multicast.enabled:
false" & "node.master=true" so i am creating indices via java API, . i
kill(Kill -9 ) the elasticsearch immediately after the index is created.
when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4
shards in a corrupt state, with "503" status code.

On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey,

can you reliably recreate this? And try to create a gist? Preferrably,
when using elasticsearch 0.90.9. When you create an index, you usually
create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
and do not use the defaults (are you changing the defaults somewhere else
as well)? It would be great if you could provide a reproducible example
using a gist, like mentioned in http://www.elasticsearch.org/help

--Alex

On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer tarang.dawer@gmail.comwrote:

Hi all
I am facing an issue of corrupt index creation, whenever the es node is
killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.

Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQX9CgBwi%3DgWpvbidjhf%2BpqKyTg5GHjVyrh7TPe2JrshOA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(tarang dawer) #6

While browsing for this issue, i came across a comment from spinscale @


It says here, that after creating an index, it takes some time for index to
be fully functional, what i am facing as i think, is that if the node gets
killed during that time, then the index gets corrupted and indexing stops
with UnavailableShardsException being thrown.
is there some setting so that i can make sure that index creation call gets
returned only after that index is fully functional ? that would solve my
problem, as the index if created would be fully functional , if not then
the code will go again to create the index again.

On Thu, Dec 26, 2013 at 10:49 AM, Tarang Dawer tarang.dawer@gmail.comwrote:

Tried one again, this time, the status showed 3 successful shards, but
however the indexing got stuck, and after a while an exception was thrown

Caused by: org.elasticsearch.action.UnavailableShardsException:
[indexName][4] [2] shardIt, [0] active : Timeout waiting for [1m], request:
index {[indexName][indexName][ID], source["Record":"Details"]}
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)
... 3 more

and the cluster state is still in red, with elasticsearch not able to
recover the shards after the restart

Conquer Lordcluster health: red (1, 3)

On Thu, Dec 26, 2013 at 10:32 AM, Tarang Dawer tarang.dawer@gmail.comwrote:

tried this on 90.9 also . This time after the restart when i get the
index status as :-

In the elasticsearch head plugin, it shows the cluster is in red state,
with no shards being allocated to the node even after the restart of
elasticsearch.

http://localhost:9200/indexName/_status

{"ok":true,"_shards":{"total":10,"successful":0,"failed":0},"indices":{}}

In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even
after the restart of elasticsearch.

Squirrel Girlcluster health: red (1, 0)

On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer tarang.dawer@gmail.comwrote:

i have reliably recreated this many times, happens while creating index
on a single node, (default 5 shards). i have set
"action.auto_create_index: false" , "discovery.zen.ping.multicast.enabled:
false" & "node.master=true" so i am creating indices via java API, . i
kill(Kill -9 ) the elasticsearch immediately after the index is created.
when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4
shards in a corrupt state, with "503" status code.

On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey,

can you reliably recreate this? And try to create a gist? Preferrably,
when using elasticsearch 0.90.9. When you create an index, you usually
create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
and do not use the defaults (are you changing the defaults somewhere else
as well)? It would be great if you could provide a reproducible example
using a gist, like mentioned in http://www.elasticsearch.org/help

--Alex

On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer tarang.dawer@gmail.comwrote:

Hi all
I am facing an issue of corrupt index creation, whenever the es node
is killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.

Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQU9uPamQhXbZpkV%3DKvDVSJZziiEwpfDoAc%3DJqNuhA8xyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #7

You could send a waitForYellow request just after your index creation?

curl -XGET 'http://localhost:9200/_cluster/health?wait_for_status=yellow'
HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 26 déc. 2013 à 07:19, Tarang Dawer tarang.dawer@gmail.com a écrit :

While browsing for this issue, i came across a comment from spinscale @ https://github.com/elasticsearch/elasticsearch/issues/2922
It says here, that after creating an index, it takes some time for index to be fully functional, what i am facing as i think, is that if the node gets killed during that time, then the index gets corrupted and indexing stops with UnavailableShardsException being thrown.
is there some setting so that i can make sure that index creation call gets returned only after that index is fully functional ? that would solve my problem, as the index if created would be fully functional , if not then the code will go again to create the index again.

On Thu, Dec 26, 2013 at 10:49 AM, Tarang Dawer tarang.dawer@gmail.com wrote:
Tried one again, this time, the status showed 3 successful shards, but however the indexing got stuck, and after a while an exception was thrown

Caused by: org.elasticsearch.action.UnavailableShardsException: [indexName][4] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[indexName][indexName][ID], source["Record":"Details"]}
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)
... 3 more

and the cluster state is still in red, with elasticsearch not able to recover the shards after the restart

Conquer Lordcluster health: red (1, 3)

On Thu, Dec 26, 2013 at 10:32 AM, Tarang Dawer tarang.dawer@gmail.com wrote:
tried this on 90.9 also . This time after the restart when i get the index status as :-

In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even after the restart of elasticsearch.

http://localhost:9200/indexName/_status
{"ok":true,"_shards":{"total":10,"successful":0,"failed":0},"indices":{}}

In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even
after the restart of elasticsearch.

Squirrel Girlcluster health: red (1, 0)

On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer tarang.dawer@gmail.com wrote:
i have reliably recreated this many times, happens while creating index on a single node, (default 5 shards). i have set "action.auto_create_index: false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true" so i am creating indices via java API, . i kill(Kill -9 ) the elasticsearch immediately after the index is created. when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4 shards in a corrupt state, with "503" status code.

On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen alr@spinscale.de wrote:
Hey,

can you reliably recreate this? And try to create a gist? Preferrably, when using elasticsearch 0.90.9. When you create an index, you usually create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything and do not use the defaults (are you changing the defaults somewhere else as well)? It would be great if you could provide a reproducible example using a gist, like mentioned in http://www.elasticsearch.org/help

--Alex

On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer tarang.dawer@gmail.com wrote:
Hi all
I am facing an issue of corrupt index creation, whenever the es node is killed just after the index is created. When the node is restarted, the index shows 3/4 shards corrupt, with 503 status, which never recover, and as a result, my indexing gets stuck. I am doing this on a single node, with es version 90.1 . Please help me out.
Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQU9uPamQhXbZpkV%3DKvDVSJZziiEwpfDoAc%3DJqNuhA8xyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6DA8A787-3616-4F14-802C-F03BCA014063%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(tarang dawer) #8

index creation call is only made when indexmissing exception is thrown frm
elasticsearch. so in a multithreaded mode, where 1 thread goes for index
creation & creates a corrupt index, where as the other one finds the
corrupt index(no index missing exception thrown) , & eventually throws
unavailable shards exception.
for this scenario, i would be required to check waitForYellow request
before every indexing call. that would be extremely taxing on the
performance.

On Thu, Dec 26, 2013 at 1:19 PM, David Pilato david@pilato.fr wrote:

You could send a waitForYellow request just after your index creation?

curl -XGET 'http://localhost:9200/_cluster/health?wait_for_status=yellow'

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 26 déc. 2013 à 07:19, Tarang Dawer tarang.dawer@gmail.com a écrit :

While browsing for this issue, i came across a comment from spinscale @
https://github.com/elasticsearch/elasticsearch/issues/2922
It says here, that after creating an index, it takes some time for index
to be fully functional, what i am facing as i think, is that if the node
gets killed during that time, then the index gets corrupted and indexing
stops with UnavailableShardsException being thrown.
is there some setting so that i can make sure that index creation call
gets returned only after that index is fully functional ? that would solve
my problem, as the index if created would be fully functional , if not
then the code will go again to create the index again.

On Thu, Dec 26, 2013 at 10:49 AM, Tarang Dawer tarang.dawer@gmail.comwrote:

Tried one again, this time, the status showed 3 successful shards, but
however the indexing got stuck, and after a while an exception was thrown

Caused by: org.elasticsearch.action.UnavailableShardsException:
[indexName][4] [2] shardIt, [0] active : Timeout waiting for [1m], request:
index {[indexName][indexName][ID], source["Record":"Details"]}
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)
... 3 more

and the cluster state is still in red, with elasticsearch not able to
recover the shards after the restart

Conquer Lordcluster health: red (1, 3)

On Thu, Dec 26, 2013 at 10:32 AM, Tarang Dawer tarang.dawer@gmail.comwrote:

tried this on 90.9 also . This time after the restart when i get the
index status as :-

In the elasticsearch head plugin, it shows the cluster is in red state,
with no shards being allocated to the node even after the restart of
elasticsearch.

http://localhost:9200/indexName/_status

{"ok":true,"_shards":{"total":10,"successful":0,"failed":0},"indices":{}}

In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even
after the restart of elasticsearch.

Squirrel Girlcluster health: red (1, 0)

On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer tarang.dawer@gmail.comwrote:

i have reliably recreated this many times, happens while creating index
on a single node, (default 5 shards). i have set
"action.auto_create_index: false" , "discovery.zen.ping.multicast.enabled:
false" & "node.master=true" so i am creating indices via java API, . i
kill(Kill -9 ) the elasticsearch immediately after the index is created.
when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4
shards in a corrupt state, with "503" status code.

On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey,

can you reliably recreate this? And try to create a gist? Preferrably,
when using elasticsearch 0.90.9. When you create an index, you usually
create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
and do not use the defaults (are you changing the defaults somewhere else
as well)? It would be great if you could provide a reproducible example
using a gist, like mentioned in http://www.elasticsearch.org/help

--Alex

On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer tarang.dawer@gmail.comwrote:

Hi all
I am facing an issue of corrupt index creation, whenever the es node
is killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.

Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQU9uPamQhXbZpkV%3DKvDVSJZziiEwpfDoAc%3DJqNuhA8xyQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6DA8A787-3616-4F14-802C-F03BCA014063%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVfSyPYvEGG4orKDzdQ1Pzemg5n%3DZ-Sg_gaHx32Juzjow%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Clinton Gormley) #9

If you kill Elasticsearch immediately after creating the index, you
interrupt the process of shard allocation. When you restart Elasticsearch,
it assumes that the shards have been allocated somewhere and so doesn't try
to assign new shards to prevent any data loss.

The index itself isn't corrupt, it is just missing primary shards. You can
force the missing shards to be allocated using the cluster reroute API
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-reroute.html-
you will need to set allow_primary to true.

On 26 December 2013 05:36, Tarang Dawer tarang.dawer@gmail.com wrote:

i have reliably recreated this many times, happens while creating index on
a single node, (default 5 shards). i have set "action.auto_create_index:
false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true"
so i am creating indices via java API, . i kill(Kill -9 ) the elasticsearch
immediately after the index is created. when i restart the elasticsearch,
out of the 5 primary shards, it shows 3/4 shards in a corrupt state, with
"503" status code.

On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey,

can you reliably recreate this? And try to create a gist? Preferrably,
when using elasticsearch 0.90.9. When you create an index, you usually
create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
and do not use the defaults (are you changing the defaults somewhere else
as well)? It would be great if you could provide a reproducible example
using a gist, like mentioned in http://www.elasticsearch.org/help

--Alex

On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer tarang.dawer@gmail.comwrote:

Hi all
I am facing an issue of corrupt index creation, whenever the es node is
killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.

Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUrAfA6Qe_NDEQ9SwRWmOK%2BkSLY%3Dq%3D9syENAZ_yx3B5yg%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQ%2BSm_%2Bu-Amtq2X237ObROnaqWcwqnJt%3DoBEWVYC7vZGw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(tarang dawer) #10

Thanks clinton.
But shouldn't index creation be an atomic operation, and the call should
only return after the index is "properly" created, otherwise show a
indexmissing exception, or should automatically look for unassigned primary
shards for an index after the restart ?

On Thu, Dec 26, 2013 at 6:25 PM, Clinton Gormley clint@traveljury.comwrote:

If you kill Elasticsearch immediately after creating the index, you
interrupt the process of shard allocation. When you restart Elasticsearch,
it assumes that the shards have been allocated somewhere and so doesn't try
to assign new shards to prevent any data loss.

The index itself isn't corrupt, it is just missing primary shards. You
can force the missing shards to be allocated using the cluster reroute API
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-reroute.html- you will need to set allow_primary to true.

On 26 December 2013 05:36, Tarang Dawer tarang.dawer@gmail.com wrote:

i have reliably recreated this many times, happens while creating index
on a single node, (default 5 shards). i have set
"action.auto_create_index: false" , "discovery.zen.ping.multicast.enabled:
false" & "node.master=true" so i am creating indices via java API, . i
kill(Kill -9 ) the elasticsearch immediately after the index is created.
when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4
shards in a corrupt state, with "503" status code.

On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey,

can you reliably recreate this? And try to create a gist? Preferrably,
when using elasticsearch 0.90.9. When you create an index, you usually
create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
and do not use the defaults (are you changing the defaults somewhere else
as well)? It would be great if you could provide a reproducible example
using a gist, like mentioned in http://www.elasticsearch.org/help

--Alex

On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer tarang.dawer@gmail.comwrote:

Hi all
I am facing an issue of corrupt index creation, whenever the es node is
killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.

Thanks
Tarang Dawer

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUrAfA6Qe_NDEQ9SwRWmOK%2BkSLY%3Dq%3D9syENAZ_yx3B5yg%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQ%2BSm_%2Bu-Amtq2X237ObROnaqWcwqnJt%3DoBEWVYC7vZGw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUD7tZ6%2Br_JYyYfKZUXeY6-YeEpG2-EcaD1BByvHL2fJg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #11

All ES API calls are by default asynchronous and eventually consistent
(quorum).

For document indexing, you can use the refresh API call to make them
visible for search.

For index creation operation, you can add the parameter "replication=sync"
and "consistency=all" to your API call to ensure that ES will wait for all
replica shards and all nodes to complete successfully before returning.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEVVBczSkSUewW6%2Bdrov4q8MDvHGuAprmWx1iC8q-Df7A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(tarang dawer) #12

Thanks for your response Jörg.

"replication=sync" and "consistency=all" would be required to be set in the
settings while creating the index. ?

also, these settings would only be applied during index creation or would
also be applied for all the subsequent indexing operations for the
particular index created by using these settings .?

On Thu, Jan 2, 2014 at 3:12 PM, joergprante@gmail.com <joergprante@gmail.com

wrote:

All ES API calls are by default asynchronous and eventually consistent
(quorum).

For document indexing, you can use the refresh API call to make them
visible for search.

For index creation operation, you can add the parameter "replication=sync"
and "consistency=all" to your API call to ensure that ES will wait for all
replica shards and all nodes to complete successfully before returning.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEVVBczSkSUewW6%2Bdrov4q8MDvHGuAprmWx1iC8q-Df7A%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEGWFQXbDbxfxYcZSbda4MScpCBH6LMUN5iC0i7xvN2PXiNhsg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #13

Sorry my fault. I stand corrected. There is replication=sync and
consistency=all, but just for index creation that is triggered by a
document creation, and the index does not yet exist (auto creation). It's
not there for explicit index creation (where there is no document to be
created).

In case you explicitly execute index creation, you can add a master node
timeout to the operation, and if it exceeds, the operation will return that
is was not acknowledged by all nodes.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGgoJP%3DtkgSnOBHA366QgesFR4Ft5xztSrr1fX3ZWDDAw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Brian Yoder) #14

Never, never, never kill -9 and expect any application to properly and
cleanly shut down. Never.

The -9 signal cannot be caught by the process to which it is directed. The
process is ended in the middle for whatever it is doing.

Issue a normal kill, and then ES (via the JVM) will have a chance to finish
up whatever it is working on, and then shut down cleanly.

Brian

On Wednesday, December 25, 2013 11:36:32 PM UTC-5, tarang dawer wrote:

i have reliably recreated this many times, happens while creating index on
a single node, (default 5 shards). i have set "action.auto_create_index:
false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true"
so i am creating indices via java API, . i kill(Kill -9 ) the
elasticsearch immediately after the index is created.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1ba7e47f-0d9e-44fd-b1b3-628da214b499%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #15