# Corrupt index creation when elasticsearch is killed just after index is created
**URL:** https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018
**Category:** Elasticsearch
**Created:** [December 25, 2013, 3:35pm UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018 "2013-12-25T15:35:50Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@tarang\_dawer](https://discuss.elastic.co/u/tarang_dawer)
#### Post date: [December 25, 2013, 3:35pm UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/1 "2013-12-25T15:35:50Z")
Hi all
I am facing an issue of corrupt index creation, whenever the es node is
killed just after the index is created. When the node is restarted, the
index shows 3/4 shards corrupt, with 503 status, which never recover, and
as a result, my indexing gets stuck. I am doing this on a single node, with
es version 90.1 . Please help me out.
Thanks
Tarang Dawer
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [December 25, 2013, 10:28pm UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/2 "2013-12-25T22:28:11Z")
Hey,
can you reliably recreate this? And try to create a gist? Preferrably, when
using elasticsearch 0.90.9. When you create an index, you usually create 5
shards, so, how can 3/4 shards be corrupt? Did you change anything and do
not use the defaults (are you changing the defaults somewhere else as
well)? It would be great if you could provide a reproducible example using
a gist, like mentioned in [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)
--Alex
On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> Hi all
> I am facing an issue of corrupt index creation, whenever the es node is
> killed just after the index is created. When the node is restarted, the
> index shows 3/4 shards corrupt, with 503 status, which never recover, and
> as a result, my indexing gets stuck. I am doing this on a single node, with
> es version 90.1 . Please help me out.
>
> Thanks
> Tarang Dawer
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com)
> .
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj\_LwNc0ksEg-dcRDTZ-bKH\_9AqJmOLi9A4UQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@tarang\_dawer](https://discuss.elastic.co/u/tarang_dawer)
#### Post date: [December 26, 2013, 4:36am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/3 "2013-12-26T04:36:32Z")
i have reliably recreated this many times, happens while creating index on
a single node, (default 5 shards). i have set "action.auto\_create\_index:
false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true"
so i am creating indices via java API, . i kill(Kill -9 ) the elasticsearch
immediately after the index is created. when i restart the elasticsearch,
out of the 5 primary shards, it shows 3/4 shards in a corrupt state, with
"503" status code.
On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote:
> Hey,
>
> can you reliably recreate this? And try to create a gist? Preferrably,
> when using elasticsearch 0.90.9. When you create an index, you usually
> create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
> and do not use the defaults (are you changing the defaults somewhere else
> as well)? It would be great if you could provide a reproducible example
> using a gist, like mentioned in [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)
>
> --Alex
>
> On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
>
> > Hi all
> > I am facing an issue of corrupt index creation, whenever the es node is
> > killed just after the index is created. When the node is restarted, the
> > index shows 3/4 shards corrupt, with 503 status, which never recover, and
> > as a result, my indexing gets stuck. I am doing this on a single node, with
> > es version 90.1 . Please help me out.
> >
> > Thanks
> > Tarang Dawer
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj\_LwNc0ksEg-dcRDTZ-bKH\_9AqJmOLi9A4UQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com)
> .
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUrAfA6Qe\_NDEQ9SwRWmOK%2BkSLY%3Dq%3D9syENAZ\_yx3B5yg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUrAfA6Qe_NDEQ9SwRWmOK%2BkSLY%3Dq%3D9syENAZ_yx3B5yg%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@tarang\_dawer](https://discuss.elastic.co/u/tarang_dawer)
#### Post date: [December 26, 2013, 5:02am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/4 "2013-12-26T05:02:00Z")
tried this on 90.9 also . This time after the restart when i get the index
status as :-
In the elasticsearch head plugin, it shows the cluster is in red state,
with no shards being allocated to the node even after the restart of
elasticsearch.
[http://localhost:9200/indexName/\_status](http://localhost:9200/indexName/_status)
{"ok":true,"\_shards":{"total":10,"successful":0,"failed":0},"indices":{}}
In the elasticsearch head plugin, it shows the cluster is in red
state, with no shards being allocated to the node even
after the restart of elasticsearch.
Squirrel Girlcluster health: red (1, 0)
On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> i have reliably recreated this many times, happens while creating index on
> a single node, (default 5 shards). i have set "action.auto\_create\_index:
> false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true"
> so i am creating indices via java API, . i kill(Kill -9 ) the elasticsearch
> immediately after the index is created. when i restart the elasticsearch,
> out of the 5 primary shards, it shows 3/4 shards in a corrupt state, with
> "503" status code.
>
> On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de)wrote:
>
> > Hey,
> >
> > can you reliably recreate this? And try to create a gist? Preferrably,
> > when using elasticsearch 0.90.9. When you create an index, you usually
> > create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
> > and do not use the defaults (are you changing the defaults somewhere else
> > as well)? It would be great if you could provide a reproducible example
> > using a gist, like mentioned in [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)
> >
> > --Alex
> >
> > On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> >
> > > Hi all
> > > I am facing an issue of corrupt index creation, whenever the es node is
> > > killed just after the index is created. When the node is restarted, the
> > > index shows 3/4 shards corrupt, with 503 status, which never recover, and
> > > as a result, my indexing gets stuck. I am doing this on a single node, with
> > > es version 90.1 . Please help me out.
> > >
> > > Thanks
> > > Tarang Dawer
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com)
> > > .
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj\_LwNc0ksEg-dcRDTZ-bKH\_9AqJmOLi9A4UQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUZgt6nFpGgZa3arTAPJXt2ett6tqo1ytG5-XkrK\_2\_hQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUZgt6nFpGgZa3arTAPJXt2ett6tqo1ytG5-XkrK_2_hQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@tarang\_dawer](https://discuss.elastic.co/u/tarang_dawer)
#### Post date: [December 26, 2013, 5:19am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/5 "2013-12-26T05:19:14Z")
Tried one again, this time, the status showed 3 successful shards, but
however the indexing got stuck, and after a while an exception was thrown
Caused by: org.elasticsearch.action.UnavailableShardsException:
[indexName][4] [2] shardIt, [0] active : Timeout waiting for [1m], request:
index {[indexName][indexName][ID], source["Record":"Details"]}
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)
... 3 more
and the cluster state is still in red, with elasticsearch not able to
recover the shards after the restart
Conquer Lordcluster health: red (1, 3)
On Thu, Dec 26, 2013 at 10:32 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> tried this on 90.9 also . This time after the restart when i get the index
> status as :-
>
> In the elasticsearch head plugin, it shows the cluster is in red state,
> with no shards being allocated to the node even after the restart of
> elasticsearch.
>
> [http://localhost:9200/indexName/\_status](http://localhost:9200/indexName/_status)
>
> {"ok":true,"\_shards":{"total":10,"successful":0,"failed":0},"indices":{}}
>
> In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even
> after the restart of elasticsearch.
>
> Squirrel Girlcluster health: red (1, 0)
>
> On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
>
> > i have reliably recreated this many times, happens while creating index
> > on a single node, (default 5 shards). i have set
> > "action.auto\_create\_index: false" , "discovery.zen.ping.multicast.enabled:
> > false" & "node.master=true" so i am creating indices via java API, . i
> > kill(Kill -9 ) the elasticsearch immediately after the index is created.
> > when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4
> > shards in a corrupt state, with "503" status code.
> >
> > On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de)wrote:
> >
> > > Hey,
> > >
> > > can you reliably recreate this? And try to create a gist? Preferrably,
> > > when using elasticsearch 0.90.9. When you create an index, you usually
> > > create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
> > > and do not use the defaults (are you changing the defaults somewhere else
> > > as well)? It would be great if you could provide a reproducible example
> > > using a gist, like mentioned in [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)
> > >
> > > --Alex
> > >
> > > On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> > >
> > > > Hi all
> > > > I am facing an issue of corrupt index creation, whenever the es node is
> > > > killed just after the index is created. When the node is restarted, the
> > > > index shows 3/4 shards corrupt, with 503 status, which never recover, and
> > > > as a result, my indexing gets stuck. I am doing this on a single node, with
> > > > es version 90.1 . Please help me out.
> > > >
> > > > Thanks
> > > > Tarang Dawer
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com)
> > > > .
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj\_LwNc0ksEg-dcRDTZ-bKH\_9AqJmOLi9A4UQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com)
> > > .
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQX9CgBwi%3DgWpvbidjhf%2BpqKyTg5GHjVyrh7TPe2JrshOA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQX9CgBwi%3DgWpvbidjhf%2BpqKyTg5GHjVyrh7TPe2JrshOA%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@tarang\_dawer](https://discuss.elastic.co/u/tarang_dawer)
#### Post date: [December 26, 2013, 6:19am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/6 "2013-12-26T06:19:02Z")
While browsing for this issue, i came across a comment from spinscale @
>
>
> I have tried an elastic search cluster configuration with 3 nodes in Amazon EC2.… But after making the call $type-\>addDocument($doc);, I am getting "UnavailableShardsException" .
>
> Please help me to solve this issue ! Thanks in advance !
> Asuthosh
>
> The following are the configuration on each of the 3 nodes in the custer.
> \# // Node1
>
> cluster.name: MyCluster
> node.name: node1
> node.master: true
> node.data: true
> // # index.number\_of\_shards: 5 //Not confugured this, to use the default value 5
> index.number\_of\_replicas: 2
> transport.tcp.port: 9300
> http.port: 9200
> \# // Node2
>
> cluster.name: MyCluster
> node.name: node2
> node.master: false
> node.data: true
> // # index.number\_of\_shards: 5 //Not confugured this, to use the default value 5
> index.number\_of\_replicas: 2
> transport.tcp.port: 9301
> http.port: 9201
> \# // Node3
>
> cluster.name: MyCluster
> node.name: node3
> node.master: false
> node.data: true
> // # index.number\_of\_shards: 5 //Not confugured this, to use the default value 5
> index.number\_of\_replicas: 2
> transport.tcp.port: 9302
> http.port: 9202
It says here, that after creating an index, it takes some time for index to
be fully functional, what i am facing as i think, is that if the node gets
killed during that time, then the index gets corrupted and indexing stops
with UnavailableShardsException being thrown.
is there some setting so that i can make sure that index creation call gets
returned only after that index is fully functional ? that would solve my
problem, as the index if created would be fully functional , if not then
the code will go again to create the index again.
On Thu, Dec 26, 2013 at 10:49 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> Tried one again, this time, the status showed 3 successful shards, but
> however the indexing got stuck, and after a while an exception was thrown
>
> Caused by: org.elasticsearch.action.UnavailableShardsException:
> [indexName][4] [2] shardIt, [0] active : Timeout waiting for [1m], request:
> index {[indexName][indexName][ID], source["Record":"Details"]}
> at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
> at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
> at
> org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)
> ... 3 more
>
> and the cluster state is still in red, with elasticsearch not able to
> recover the shards after the restart
>
> Conquer Lordcluster health: red (1, 3)
>
> On Thu, Dec 26, 2013 at 10:32 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
>
> > tried this on 90.9 also . This time after the restart when i get the
> > index status as :-
> >
> > In the elasticsearch head plugin, it shows the cluster is in red state,
> > with no shards being allocated to the node even after the restart of
> > elasticsearch.
> >
> > [http://localhost:9200/indexName/\_status](http://localhost:9200/indexName/_status)
> >
> > {"ok":true,"\_shards":{"total":10,"successful":0,"failed":0},"indices":{}}
> >
> > In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even
> > after the restart of elasticsearch.
> >
> > Squirrel Girlcluster health: red (1, 0)
> >
> > On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> >
> > > i have reliably recreated this many times, happens while creating index
> > > on a single node, (default 5 shards). i have set
> > > "action.auto\_create\_index: false" , "discovery.zen.ping.multicast.enabled:
> > > false" & "node.master=true" so i am creating indices via java API, . i
> > > kill(Kill -9 ) the elasticsearch immediately after the index is created.
> > > when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4
> > > shards in a corrupt state, with "503" status code.
> > >
> > > On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de)wrote:
> > >
> > > > Hey,
> > > >
> > > > can you reliably recreate this? And try to create a gist? Preferrably,
> > > > when using elasticsearch 0.90.9. When you create an index, you usually
> > > > create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
> > > > and do not use the defaults (are you changing the defaults somewhere else
> > > > as well)? It would be great if you could provide a reproducible example
> > > > using a gist, like mentioned in [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)
> > > >
> > > > --Alex
> > > >
> > > > On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> > > >
> > > > > Hi all
> > > > > I am facing an issue of corrupt index creation, whenever the es node
> > > > > is killed just after the index is created. When the node is restarted, the
> > > > > index shows 3/4 shards corrupt, with 503 status, which never recover, and
> > > > > as a result, my indexing gets stuck. I am doing this on a single node, with
> > > > > es version 90.1 . Please help me out.
> > > > >
> > > > > Thanks
> > > > > Tarang Dawer
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > To view this discussion on the web visit
> > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com)
> > > > > .
> > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj\_LwNc0ksEg-dcRDTZ-bKH\_9AqJmOLi9A4UQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com)
> > > > .
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQU9uPamQhXbZpkV%3DKvDVSJZziiEwpfDoAc%3DJqNuhA8xyQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQU9uPamQhXbZpkV%3DKvDVSJZziiEwpfDoAc%3DJqNuhA8xyQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [December 26, 2013, 7:49am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/7 "2013-12-26T07:49:27Z")
You could send a waitForYellow request just after your index creation?
curl -XGET '[http://localhost:9200/\_cluster/health?wait\_for\_status=yellow](http://localhost:9200/_cluster/health?wait_for_status=yellow)'
HTH
--
David 😉
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
Le 26 déc. 2013 à 07:19, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com) a écrit :
While browsing for this issue, i came across a comment from spinscale @ [UnavailableShardsException in elastic search cluster configuration ! · Issue #2922 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2922)
It says here, that after creating an index, it takes some time for index to be fully functional, what i am facing as i think, is that if the node gets killed during that time, then the index gets corrupted and indexing stops with UnavailableShardsException being thrown.
is there some setting so that i can make sure that index creation call gets returned only after that index is fully functional ? that would solve my problem, as the index if created would be fully functional , if not then the code will go again to create the index again.
> On Thu, Dec 26, 2013 at 10:49 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com) wrote:
> Tried one again, this time, the status showed 3 successful shards, but however the indexing got stuck, and after a while an exception was thrown
>
> Caused by: org.elasticsearch.action.UnavailableShardsException: [indexName][4] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[indexName][indexName][ID], source["Record":"Details"]}
> at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
> at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
> at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)
> ... 3 more
>
> and the cluster state is still in red, with elasticsearch not able to recover the shards after the restart
>
> Conquer Lordcluster health: red (1, 3)
>
> > On Thu, Dec 26, 2013 at 10:32 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com) wrote:
> > tried this on 90.9 also . This time after the restart when i get the index status as :-
> >
> > In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even after the restart of elasticsearch.
> >
> > [http://localhost:9200/indexName/\_status](http://localhost:9200/indexName/_status)
> > {"ok":true,"\_shards":{"total":10,"successful":0,"failed":0},"indices":{}}
> >
> > In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even
> > after the restart of elasticsearch.
> >
> > Squirrel Girlcluster health: red (1, 0)
> >
> > > On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com) wrote:
> > > i have reliably recreated this many times, happens while creating index on a single node, (default 5 shards). i have set "action.auto\_create\_index: false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true" so i am creating indices via java API, . i kill(Kill -9 ) the elasticsearch immediately after the index is created. when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4 shards in a corrupt state, with "503" status code.
> > >
> > > > On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote:
> > > > Hey,
> > > >
> > > > can you reliably recreate this? And try to create a gist? Preferrably, when using elasticsearch 0.90.9. When you create an index, you usually create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything and do not use the defaults (are you changing the defaults somewhere else as well)? It would be great if you could provide a reproducible example using a gist, like mentioned in [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)
> > > >
> > > > --Alex
> > > >
> > > > > On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com) wrote:
> > > > > Hi all
> > > > > I am facing an issue of corrupt index creation, whenever the es node is killed just after the index is created. When the node is restarted, the index shows 3/4 shards corrupt, with 503 status, which never recover, and as a result, my indexing gets stuck. I am doing this on a single node, with es version 90.1 . Please help me out.
> > > > > Thanks
> > > > > Tarang Dawer
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com).
> > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj\_LwNc0ksEg-dcRDTZ-bKH\_9AqJmOLi9A4UQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com).
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQU9uPamQhXbZpkV%3DKvDVSJZziiEwpfDoAc%3DJqNuhA8xyQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQU9uPamQhXbZpkV%3DKvDVSJZziiEwpfDoAc%3DJqNuhA8xyQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6DA8A787-3616-4F14-802C-F03BCA014063%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/6DA8A787-3616-4F14-802C-F03BCA014063%40pilato.fr).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@tarang\_dawer](https://discuss.elastic.co/u/tarang_dawer)
#### Post date: [December 26, 2013, 10:56am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/8 "2013-12-26T10:56:58Z")
index creation call is only made when indexmissing exception is thrown frm
elasticsearch. so in a multithreaded mode, where 1 thread goes for index
creation & creates a corrupt index, where as the other one finds the
corrupt index(no index missing exception thrown) , & eventually throws
unavailable shards exception.
for this scenario, i would be required to check waitForYellow request
before every indexing call. that would be extremely taxing on the
performance.
On Thu, Dec 26, 2013 at 1:19 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote:
> You could send a waitForYellow request just after your index creation?
>
> curl -XGET '[http://localhost:9200/\_cluster/health?wait\_for\_status=yellow](http://localhost:9200/_cluster/health?wait_for_status=yellow)'
>
> HTH
>
> --
> David 😉
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> Le 26 déc. 2013 à 07:19, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com) a écrit :
>
> While browsing for this issue, i came across a comment from spinscale @
> [UnavailableShardsException in elastic search cluster configuration ! · Issue #2922 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2922)
> It says here, that after creating an index, it takes some time for index
> to be fully functional, what i am facing as i think, is that if the node
> gets killed during that time, then the index gets corrupted and indexing
> stops with UnavailableShardsException being thrown.
> is there some setting so that i can make sure that index creation call
> gets returned only after that index is fully functional ? that would solve
> my problem, as the index if created would be fully functional , if not
> then the code will go again to create the index again.
>
> On Thu, Dec 26, 2013 at 10:49 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
>
> > Tried one again, this time, the status showed 3 successful shards, but
> > however the indexing got stuck, and after a while an exception was thrown
> >
> > Caused by: org.elasticsearch.action.UnavailableShardsException:
> > [indexName][4] [2] shardIt, [0] active : Timeout waiting for [1m], request:
> > index {[indexName][indexName][ID], source["Record":"Details"]}
> > at
> > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
> > at
> > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
> > at
> > org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)
> > ... 3 more
> >
> > and the cluster state is still in red, with elasticsearch not able to
> > recover the shards after the restart
> >
> > Conquer Lordcluster health: red (1, 3)
> >
> > On Thu, Dec 26, 2013 at 10:32 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> >
> > > tried this on 90.9 also . This time after the restart when i get the
> > > index status as :-
> > >
> > > In the elasticsearch head plugin, it shows the cluster is in red state,
> > > with no shards being allocated to the node even after the restart of
> > > elasticsearch.
> > >
> > > [http://localhost:9200/indexName/\_status](http://localhost:9200/indexName/_status)
> > >
> > > {"ok":true,"\_shards":{"total":10,"successful":0,"failed":0},"indices":{}}
> > >
> > > In the elasticsearch head plugin, it shows the cluster is in red state, with no shards being allocated to the node even
> > > after the restart of elasticsearch.
> > >
> > > Squirrel Girlcluster health: red (1, 0)
> > >
> > > On Thu, Dec 26, 2013 at 10:06 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> > >
> > > > i have reliably recreated this many times, happens while creating index
> > > > on a single node, (default 5 shards). i have set
> > > > "action.auto\_create\_index: false" , "discovery.zen.ping.multicast.enabled:
> > > > false" & "node.master=true" so i am creating indices via java API, . i
> > > > kill(Kill -9 ) the elasticsearch immediately after the index is created.
> > > > when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4
> > > > shards in a corrupt state, with "503" status code.
> > > >
> > > > On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de)wrote:
> > > >
> > > > > Hey,
> > > > >
> > > > > can you reliably recreate this? And try to create a gist? Preferrably,
> > > > > when using elasticsearch 0.90.9. When you create an index, you usually
> > > > > create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
> > > > > and do not use the defaults (are you changing the defaults somewhere else
> > > > > as well)? It would be great if you could provide a reproducible example
> > > > > using a gist, like mentioned in [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)
> > > > >
> > > > > --Alex
> > > > >
> > > > > On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> > > > >
> > > > > > Hi all
> > > > > > I am facing an issue of corrupt index creation, whenever the es node
> > > > > > is killed just after the index is created. When the node is restarted, the
> > > > > > index shows 3/4 shards corrupt, with 503 status, which never recover, and
> > > > > > as a result, my indexing gets stuck. I am doing this on a single node, with
> > > > > > es version 90.1 . Please help me out.
> > > > > >
> > > > > > Thanks
> > > > > > Tarang Dawer
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > > To view this discussion on the web visit
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com)
> > > > > > .
> > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > To view this discussion on the web visit
> > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj\_LwNc0ksEg-dcRDTZ-bKH\_9AqJmOLi9A4UQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com)
> > > > > .
> > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQU9uPamQhXbZpkV%3DKvDVSJZziiEwpfDoAc%3DJqNuhA8xyQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQU9uPamQhXbZpkV%3DKvDVSJZziiEwpfDoAc%3DJqNuhA8xyQ%40mail.gmail.com)
> .
>
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/6DA8A787-3616-4F14-802C-F03BCA014063%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/6DA8A787-3616-4F14-802C-F03BCA014063%40pilato.fr)
> .
>
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVfSyPYvEGG4orKDzdQ1Pzemg5n%3DZ-Sg\_gaHx32Juzjow%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVfSyPYvEGG4orKDzdQ1Pzemg5n%3DZ-Sg_gaHx32Juzjow%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [December 26, 2013, 12:55pm UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/9 "2013-12-26T12:55:12Z")
If you kill Elasticsearch immediately after creating the index, you
interrupt the process of shard allocation. When you restart Elasticsearch,
it assumes that the shards have been allocated somewhere and so doesn't try
to assign new shards to prevent any data loss.
The index itself isn't corrupt, it is just missing primary shards. You can
force the missing shards to be allocated using the cluster reroute API
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
you will need to set allow\_primary to true.
On 26 December 2013 05:36, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com) wrote:
> i have reliably recreated this many times, happens while creating index on
> a single node, (default 5 shards). i have set "action.auto\_create\_index:
> false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true"
> so i am creating indices via java API, . i kill(Kill -9 ) the elasticsearch
> immediately after the index is created. when i restart the elasticsearch,
> out of the 5 primary shards, it shows 3/4 shards in a corrupt state, with
> "503" status code.
>
> On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de)wrote:
>
> > Hey,
> >
> > can you reliably recreate this? And try to create a gist? Preferrably,
> > when using elasticsearch 0.90.9. When you create an index, you usually
> > create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
> > and do not use the defaults (are you changing the defaults somewhere else
> > as well)? It would be great if you could provide a reproducible example
> > using a gist, like mentioned in [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)
> >
> > --Alex
> >
> > On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> >
> > > Hi all
> > > I am facing an issue of corrupt index creation, whenever the es node is
> > > killed just after the index is created. When the node is restarted, the
> > > index shows 3/4 shards corrupt, with 503 status, which never recover, and
> > > as a result, my indexing gets stuck. I am doing this on a single node, with
> > > es version 90.1 . Please help me out.
> > >
> > > Thanks
> > > Tarang Dawer
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com)
> > > .
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj\_LwNc0ksEg-dcRDTZ-bKH\_9AqJmOLi9A4UQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUrAfA6Qe\_NDEQ9SwRWmOK%2BkSLY%3Dq%3D9syENAZ\_yx3B5yg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUrAfA6Qe_NDEQ9SwRWmOK%2BkSLY%3Dq%3D9syENAZ_yx3B5yg%40mail.gmail.com)
> .
>
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQ%2BSm\_%2Bu-Amtq2X237ObROnaqWcwqnJt%3DoBEWVYC7vZGw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQ%2BSm_%2Bu-Amtq2X237ObROnaqWcwqnJt%3DoBEWVYC7vZGw%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@tarang\_dawer](https://discuss.elastic.co/u/tarang_dawer)
#### Post date: [January 2, 2014, 7:59am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/10 "2014-01-02T07:59:58Z")
Thanks clinton.
But shouldn't index creation be an atomic operation, and the call should
only return after the index is "properly" created, otherwise show a
indexmissing exception, or should automatically look for unassigned primary
shards for an index after the restart ?
On Thu, Dec 26, 2013 at 6:25 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote:
> If you kill Elasticsearch immediately after creating the index, you
> interrupt the process of shard allocation. When you restart Elasticsearch,
> it assumes that the shards have been allocated somewhere and so doesn't try
> to assign new shards to prevent any data loss.
>
> The index itself isn't corrupt, it is just missing primary shards. You
> can force the missing shards to be allocated using the cluster reroute API
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-reroute.html-) you will need to set allow\_primary to true.
>
> On 26 December 2013 05:36, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com) wrote:
>
> > i have reliably recreated this many times, happens while creating index
> > on a single node, (default 5 shards). i have set
> > "action.auto\_create\_index: false" , "discovery.zen.ping.multicast.enabled:
> > false" & "node.master=true" so i am creating indices via java API, . i
> > kill(Kill -9 ) the elasticsearch immediately after the index is created.
> > when i restart the elasticsearch, out of the 5 primary shards, it shows 3/4
> > shards in a corrupt state, with "503" status code.
> >
> > On Thu, Dec 26, 2013 at 3:58 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de)wrote:
> >
> > > Hey,
> > >
> > > can you reliably recreate this? And try to create a gist? Preferrably,
> > > when using elasticsearch 0.90.9. When you create an index, you usually
> > > create 5 shards, so, how can 3/4 shards be corrupt? Did you change anything
> > > and do not use the defaults (are you changing the defaults somewhere else
> > > as well)? It would be great if you could provide a reproducible example
> > > using a gist, like mentioned in [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)
> > >
> > > --Alex
> > >
> > > On Wed, Dec 25, 2013 at 4:35 PM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote:
> > >
> > > > Hi all
> > > > I am facing an issue of corrupt index creation, whenever the es node is
> > > > killed just after the index is created. When the node is restarted, the
> > > > index shows 3/4 shards corrupt, with 503 status, which never recover, and
> > > > as a result, my indexing gets stuck. I am doing this on a single node, with
> > > > es version 90.1 . Please help me out.
> > > >
> > > > Thanks
> > > > Tarang Dawer
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQVihK%3DzJ9wE%2BzufhOSoOHg97g-FM6FFnqtw8JCpYO4VCQ%40mail.gmail.com)
> > > > .
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj\_LwNc0ksEg-dcRDTZ-bKH\_9AqJmOLi9A4UQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-ggGZXBRj_LwNc0ksEg-dcRDTZ-bKH_9AqJmOLi9A4UQ%40mail.gmail.com)
> > > .
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUrAfA6Qe\_NDEQ9SwRWmOK%2BkSLY%3Dq%3D9syENAZ\_yx3B5yg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUrAfA6Qe_NDEQ9SwRWmOK%2BkSLY%3Dq%3D9syENAZ_yx3B5yg%40mail.gmail.com)
> > .
> >
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQ%2BSm\_%2Bu-Amtq2X237ObROnaqWcwqnJt%3DoBEWVYC7vZGw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQ%2BSm_%2Bu-Amtq2X237ObROnaqWcwqnJt%3DoBEWVYC7vZGw%40mail.gmail.com)
> .
>
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUD7tZ6%2Br\_JYyYfKZUXeY6-YeEpG2-EcaD1BByvHL2fJg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQUD7tZ6%2Br_JYyYfKZUXeY6-YeEpG2-EcaD1BByvHL2fJg%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [January 2, 2014, 9:42am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/11 "2014-01-02T09:42:39Z")
All ES API calls are by default asynchronous and eventually consistent
(quorum).
For document indexing, you can use the refresh API call to make them
visible for search.
For index creation operation, you can add the parameter "replication=sync"
and "consistency=all" to your API call to ensure that ES will wait for all
replica shards and all nodes to complete successfully before returning.
Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEVVBczSkSUewW6%2Bdrov4q8MDvHGuAprmWx1iC8q-Df7A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEVVBczSkSUewW6%2Bdrov4q8MDvHGuAprmWx1iC8q-Df7A%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@tarang\_dawer](https://discuss.elastic.co/u/tarang_dawer)
#### Post date: [January 9, 2014, 5:29am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/12 "2014-01-09T05:29:05Z")
Thanks for your response Jörg.
"replication=sync" and "consistency=all" would be required to be set in the
settings while creating the index. ?
also, these settings would only be applied during index creation or would
also be applied for all the subsequent indexing operations for the
particular index created by using these settings .?
On Thu, Jan 2, 2014 at 3:12 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \<[joergprante@gmail.com](mailto:joergprante@gmail.com)
> wrote:
> All ES API calls are by default asynchronous and eventually consistent
> (quorum).
>
> For document indexing, you can use the refresh API call to make them
> visible for search.
>
> For index creation operation, you can add the parameter "replication=sync"
> and "consistency=all" to your API call to ensure that ES will wait for all
> replica shards and all nodes to complete successfully before returning.
>
> Jörg
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEVVBczSkSUewW6%2Bdrov4q8MDvHGuAprmWx1iC8q-Df7A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEVVBczSkSUewW6%2Bdrov4q8MDvHGuAprmWx1iC8q-Df7A%40mail.gmail.com)
> .
>
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEGWFQXbDbxfxYcZSbda4MScpCBH6LMUN5iC0i7xvN2PXiNhsg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEGWFQXbDbxfxYcZSbda4MScpCBH6LMUN5iC0i7xvN2PXiNhsg%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [January 9, 2014, 3:16pm UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/13 "2014-01-09T15:16:52Z")
Sorry my fault. I stand corrected. There is replication=sync and
consistency=all, but just for index creation that is triggered by a
document creation, and the index does not yet exist (auto creation). It's
not there for explicit index creation (where there is no document to be
created).
In case you explicitly execute index creation, you can add a master node
timeout to the operation, and if it exceeds, the operation will return that
is was not acknowledged by all nodes.
Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGgoJP%3DtkgSnOBHA366QgesFR4Ft5xztSrr1fX3ZWDDAw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGgoJP%3DtkgSnOBHA366QgesFR4Ft5xztSrr1fX3ZWDDAw%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)
#### Post date: [January 9, 2014, 5:17pm UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/14 "2014-01-09T17:17:52Z")
_Never, never, never_ kill -9 and expect any application to properly and
cleanly shut down. Never.
The -9 signal cannot be caught by the process to which it is directed. The
process is ended in the middle for whatever it is doing.
Issue a normal kill, and then ES (via the JVM) will have a chance to finish
up whatever it is working on, and then shut down cleanly.
Brian
On Wednesday, December 25, 2013 11:36:32 PM UTC-5, tarang dawer wrote:
> i have reliably recreated this many times, happens while creating index on
> a single node, (default 5 shards). i have set "action.auto\_create\_index:
> false" , "discovery.zen.ping.multicast.enabled: false" & "node.master=true"
> so i am creating indices via java API, . i kill(_Kill -9_ ) the
> elasticsearch immediately after the index is created.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1ba7e47f-0d9e-44fd-b1b3-628da214b499%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1ba7e47f-0d9e-44fd-b1b3-628da214b499%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:57am UTC](https://discuss.elastic.co/t/corrupt-index-creation-when-elasticsearch-is-killed-just-after-index-is-created/15018/15 "2017-07-06T01:57:39Z")