New index immediately becomes red


(Andreas Echavez) #1

Hi,

I'm seeing some problems with a two-node cluster where all my shards have
gone Yellow, and new shards are refusing to allocate.
Both running 90.5 & Java 1.7.0_25

I've tried almost everything I can think of to get my existing indexes
passed initialization (routing API, reducing and increasing replicas,
restarting and clearing out nodes) but no luck. Any ideas on how to
troubleshoot this one?

curl -XPUT 'http://localhost:9200/test_index/'

{"ok":true,"acknowledged":false}

curl -s -XGET

'http://localhost:9200/_cluster/health?pretty=true&level=indices' | grep
test_index -A15
"test_index" : {
"status" : "red",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 3,
"unassigned_shards" : 7
},

Shards seem to be stuck in initializing...

Node (os) stats
https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt

Cluster health
https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt

https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(vaidik) #2

How many shards do you have on your node (i.e. no. of indices * no. of
shards * no. of replicas) ? I was seeing a similar issues when I created a
lot of indices with a high number of shards on a 2 node cluster. In that
case, it is probably because of limited file descriptors. But that's my
assumption and I am fairly new to this world.

Vaidik Kapoor
vaidikkapoor.info

On 20 November 2013 01:55, Andreas Echavez oceanplexian@gmail.com wrote:

Hi,

I'm seeing some problems with a two-node cluster where all my shards have
gone Yellow, and new shards are refusing to allocate.
Both running 90.5 & Java 1.7.0_25

I've tried almost everything I can think of to get my existing indexes
passed initialization (routing API, reducing and increasing replicas,
restarting and clearing out nodes) but no luck. Any ideas on how to
troubleshoot this one?

curl -XPUT 'http://localhost:9200/test_index/'

{"ok":true,"acknowledged":false}

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' | grep
test_index -A15
"test_index" : {
"status" : "red",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 3,
"unassigned_shards" : 7
},

Shards seem to be stuck in initializing...

Node (os) stats

https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt

Cluster health

https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt

https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Kurt Hurtado) #3

Hi Andreas,
You said you reduced and increase replicas... have you also experimented
with number of shards?
Best,
Kurt

On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote:

How many shards do you have on your node (i.e. no. of indices * no. of
shards * no. of replicas) ? I was seeing a similar issues when I created a
lot of indices with a high number of shards on a 2 node cluster. In that
case, it is probably because of limited file descriptors. But that's my
assumption and I am fairly new to this world.

Vaidik Kapoor
vaidikkapoor.info

On 20 November 2013 01:55, Andreas Echavez <oceanp...@gmail.com<javascript:>

wrote:

Hi,

I'm seeing some problems with a two-node cluster where all my shards have
gone Yellow, and new shards are refusing to allocate.
Both running 90.5 & Java 1.7.0_25

I've tried almost everything I can think of to get my existing indexes
passed initialization (routing API, reducing and increasing replicas,
restarting and clearing out nodes) but no luck. Any ideas on how to
troubleshoot this one?

curl -XPUT 'http://localhost:9200/test_index/'

{"ok":true,"acknowledged":false}

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' | grep
test_index -A15
"test_index" : {
"status" : "red",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 3,
"unassigned_shards" : 7
},

Shards seem to be stuck in initializing...

Node (os) stats

https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt

Cluster health

https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt

https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Andreas Echavez) #4

ATM it is set to 300k file descriptors which should cover it. I've got
about ~220 shards across ~25 indexes or so.

Thanks for the advice though, I did just double check it to be sure.

https://gist.github.com/oceanplexian/70f243b8482016174ce1/raw/b91dec063e9b3a548acf261487a0de29b6a7ce7b/gistfile1.txt

Thanks,
Andreas

On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote:

How many shards do you have on your node (i.e. no. of indices * no. of
shards * no. of replicas) ? I was seeing a similar issues when I created a
lot of indices with a high number of shards on a 2 node cluster. In that
case, it is probably because of limited file descriptors. But that's my
assumption and I am fairly new to this world.

Vaidik Kapoor
vaidikkapoor.info

On 20 November 2013 01:55, Andreas Echavez <oceanp...@gmail.com<javascript:>

wrote:

Hi,

I'm seeing some problems with a two-node cluster where all my shards have
gone Yellow, and new shards are refusing to allocate.
Both running 90.5 & Java 1.7.0_25

I've tried almost everything I can think of to get my existing indexes
passed initialization (routing API, reducing and increasing replicas,
restarting and clearing out nodes) but no luck. Any ideas on how to
troubleshoot this one?

curl -XPUT 'http://localhost:9200/test_index/'

{"ok":true,"acknowledged":false}

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' | grep
test_index -A15
"test_index" : {
"status" : "red",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 3,
"unassigned_shards" : 7
},

Shards seem to be stuck in initializing...

Node (os) stats

https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt

Cluster health

https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt

https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Andreas Echavez) #5

Trying with 1 shard and 0 replicas seems to be the same strange behavior

curl -XPUT 'http://localhost:9200/test_index4/' -d '{

 "settings" : {
     "index" : {
         "number_of_shards" : 1,
         "number_of_replicas" : 0
     }
 }

}'
{"ok":true,"acknowledged":false}root@es4:/data/elasticsearch#

curl -s -XGET

'http://localhost:9200/_cluster/health?pretty=true&level=indices' | grep
test_index4 -A10
"test_index4" : {
"status" : "red",
"number_of_shards" : 1,
"number_of_replicas" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},

Thanks,
Andreas

On Tuesday, November 19, 2013 12:37:00 PM UTC-8, Kurt Hurtado wrote:

Hi Andreas,
You said you reduced and increase replicas... have you also experimented
with number of shards?
Best,
Kurt

On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote:

How many shards do you have on your node (i.e. no. of indices * no. of
shards * no. of replicas) ? I was seeing a similar issues when I created a
lot of indices with a high number of shards on a 2 node cluster. In that
case, it is probably because of limited file descriptors. But that's my
assumption and I am fairly new to this world.

Vaidik Kapoor
vaidikkapoor.info

On 20 November 2013 01:55, Andreas Echavez oceanp...@gmail.com wrote:

Hi,

I'm seeing some problems with a two-node cluster where all my shards
have gone Yellow, and new shards are refusing to allocate.
Both running 90.5 & Java 1.7.0_25

I've tried almost everything I can think of to get my existing indexes
passed initialization (routing API, reducing and increasing replicas,
restarting and clearing out nodes) but no luck. Any ideas on how to
troubleshoot this one?

curl -XPUT 'http://localhost:9200/test_index/'

{"ok":true,"acknowledged":false}

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' |
grep test_index -A15
"test_index" : {
"status" : "red",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 3,
"unassigned_shards" : 7
},

Shards seem to be stuck in initializing...

Node (os) stats

https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt

Cluster health

https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt

https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Kurt Hurtado) #6

So (assuming) you have time-based indices, do you close them once they are
no longer being written to?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-open-close.html

On Tuesday, November 19, 2013 12:45:00 PM UTC-8, Andreas Echavez wrote:

Trying with 1 shard and 0 replicas seems to be the same strange behavior

curl -XPUT 'http://localhost:9200/test_index4/' -d '{

 "settings" : {
     "index" : {
         "number_of_shards" : 1,
         "number_of_replicas" : 0
     }
 }

}'
{"ok":true,"acknowledged":false}root@es4:/data/elasticsearch#

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' | grep
test_index4 -A10
"test_index4" : {
"status" : "red",
"number_of_shards" : 1,
"number_of_replicas" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},

Thanks,
Andreas

On Tuesday, November 19, 2013 12:37:00 PM UTC-8, Kurt Hurtado wrote:

Hi Andreas,
You said you reduced and increase replicas... have you also experimented
with number of shards?
Best,
Kurt

On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote:

How many shards do you have on your node (i.e. no. of indices * no. of
shards * no. of replicas) ? I was seeing a similar issues when I created a
lot of indices with a high number of shards on a 2 node cluster. In that
case, it is probably because of limited file descriptors. But that's my
assumption and I am fairly new to this world.

Vaidik Kapoor
vaidikkapoor.info

On 20 November 2013 01:55, Andreas Echavez oceanp...@gmail.com wrote:

Hi,

I'm seeing some problems with a two-node cluster where all my shards
have gone Yellow, and new shards are refusing to allocate.
Both running 90.5 & Java 1.7.0_25

I've tried almost everything I can think of to get my existing indexes
passed initialization (routing API, reducing and increasing replicas,
restarting and clearing out nodes) but no luck. Any ideas on how to
troubleshoot this one?

curl -XPUT 'http://localhost:9200/test_index/'

{"ok":true,"acknowledged":false}

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' |
grep test_index -A15
"test_index" : {
"status" : "red",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 3,
"unassigned_shards" : 7
},

Shards seem to be stuck in initializing...

Node (os) stats

https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt

Cluster health

https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt

https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Andreas Echavez) #7

Unfortunately I don't have time based indices nor do we open/close them.
Interestingly enough (for the sake of testing) it refuses to let me close
an index because the primary isn't allocated

{"error":"RemoteTransportException[[elasticsearch-prod-5][inet[/10.156.208.43:9300]][indices/close]];

nested: IndexPrimaryShardNotAllocatedException[[test_index4] primary not
allocated post api]; ","status":409}

If anyone had any other ideas it'd be greatly appreciated. It really feels
like a disk full or out of memory problem, but writing to existing indexes
works fine, just not creating new ones or routing shards.

Thanks,
Andreas

On Tuesday, November 19, 2013 12:57:50 PM UTC-8, Kurt Hurtado wrote:

So (assuming) you have time-based indices, do you close them once they are
no longer being written to?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-open-close.html

On Tuesday, November 19, 2013 12:45:00 PM UTC-8, Andreas Echavez wrote:

Trying with 1 shard and 0 replicas seems to be the same strange behavior

curl -XPUT 'http://localhost:9200/test_index4/' -d '{

 "settings" : {
     "index" : {
         "number_of_shards" : 1,
         "number_of_replicas" : 0
     }
 }

}'
{"ok":true,"acknowledged":false}root@es4:/data/elasticsearch#

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' | grep
test_index4 -A10
"test_index4" : {
"status" : "red",
"number_of_shards" : 1,
"number_of_replicas" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},

Thanks,
Andreas

On Tuesday, November 19, 2013 12:37:00 PM UTC-8, Kurt Hurtado wrote:

Hi Andreas,
You said you reduced and increase replicas... have you also experimented
with number of shards?
Best,
Kurt

On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote:

How many shards do you have on your node (i.e. no. of indices * no. of
shards * no. of replicas) ? I was seeing a similar issues when I created a
lot of indices with a high number of shards on a 2 node cluster. In that
case, it is probably because of limited file descriptors. But that's my
assumption and I am fairly new to this world.

Vaidik Kapoor
vaidikkapoor.info

On 20 November 2013 01:55, Andreas Echavez oceanp...@gmail.com wrote:

Hi,

I'm seeing some problems with a two-node cluster where all my shards
have gone Yellow, and new shards are refusing to allocate.
Both running 90.5 & Java 1.7.0_25

I've tried almost everything I can think of to get my existing indexes
passed initialization (routing API, reducing and increasing replicas,
restarting and clearing out nodes) but no luck. Any ideas on how to
troubleshoot this one?

curl -XPUT 'http://localhost:9200/test_index/'

{"ok":true,"acknowledged":false}

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' |
grep test_index -A15
"test_index" : {
"status" : "red",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 3,
"unassigned_shards" : 7
},

Shards seem to be stuck in initializing...

Node (os) stats

https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt

Cluster health

https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt

https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Andreas Echavez) #8

So I figured this out for anyone encountering a similar issue:

Looked through the logs and found this error being displayed when ES tried
to create some indexes:

Caused by: org.elasticsearch.env.FailedToResolveConfigException: Failed to

resolve config path [/etc/elasticsearch/synonyms/location_synonyms.txt],
tried file path [/etc/elasticsearch/synonyms/location_synonyms.txt], path
file [/etc/elasticsearch/etc/elasticsearch/synonyms/location_synonyms.txt],
and classpath

Looks like the term synonyms file was blocking shards from being allocated.
Even though the files were in the listed locations (note they are relative
to the conf dir) with all permissions verified, I ended up having to dump
them in the classpath for ES to finally pick them up. That was the trick.

Everything is back to normal and recovering.

Thanks!
Andreas

On Tuesday, November 19, 2013 2:11:58 PM UTC-8, Andreas Echavez wrote:

Unfortunately I don't have time based indices nor do we open/close them.
Interestingly enough (for the sake of testing) it refuses to let me close
an index because the primary isn't allocated

{"error":"RemoteTransportException[[elasticsearch-prod-5][inet[/10.156.208.43:9300]][indices/close]];

nested: IndexPrimaryShardNotAllocatedException[[test_index4] primary not
allocated post api]; ","status":409}

If anyone had any other ideas it'd be greatly appreciated. It really feels
like a disk full or out of memory problem, but writing to existing indexes
works fine, just not creating new ones or routing shards.

Thanks,
Andreas

On Tuesday, November 19, 2013 12:57:50 PM UTC-8, Kurt Hurtado wrote:

So (assuming) you have time-based indices, do you close them once they
are no longer being written to?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-open-close.html

On Tuesday, November 19, 2013 12:45:00 PM UTC-8, Andreas Echavez wrote:

Trying with 1 shard and 0 replicas seems to be the same strange behavior

curl -XPUT 'http://localhost:9200/test_index4/' -d '{

 "settings" : {
     "index" : {
         "number_of_shards" : 1,
         "number_of_replicas" : 0
     }
 }

}'
{"ok":true,"acknowledged":false}root@es4:/data/elasticsearch#

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' |
grep test_index4 -A10
"test_index4" : {
"status" : "red",
"number_of_shards" : 1,
"number_of_replicas" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},

Thanks,
Andreas

On Tuesday, November 19, 2013 12:37:00 PM UTC-8, Kurt Hurtado wrote:

Hi Andreas,
You said you reduced and increase replicas... have you also
experimented with number of shards?
Best,
Kurt

On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote:

How many shards do you have on your node (i.e. no. of indices * no. of
shards * no. of replicas) ? I was seeing a similar issues when I created a
lot of indices with a high number of shards on a 2 node cluster. In that
case, it is probably because of limited file descriptors. But that's my
assumption and I am fairly new to this world.

Vaidik Kapoor
vaidikkapoor.info

On 20 November 2013 01:55, Andreas Echavez oceanp...@gmail.comwrote:

Hi,

I'm seeing some problems with a two-node cluster where all my shards
have gone Yellow, and new shards are refusing to allocate.
Both running 90.5 & Java 1.7.0_25

I've tried almost everything I can think of to get my existing
indexes passed initialization (routing API, reducing and increasing
replicas, restarting and clearing out nodes) but no luck. Any ideas on how
to troubleshoot this one?

curl -XPUT 'http://localhost:9200/test_index/'

{"ok":true,"acknowledged":false}

curl -s -XGET '

http://localhost:9200/_cluster/health?pretty=true&level=indices' |
grep test_index -A15
"test_index" : {
"status" : "red",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 3,
"unassigned_shards" : 7
},

Shards seem to be stuck in initializing...

Node (os) stats

https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt

Cluster health

https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt

https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #9