So I figured this out for anyone encountering a similar issue:
Looked through the logs and found this error being displayed when ES tried
to create some indexes:
Caused by: org.elasticsearch.env.FailedToResolveConfigException: Failed to
resolve config path [/etc/elasticsearch/synonyms/location_synonyms.txt],
tried file path [/etc/elasticsearch/synonyms/location_synonyms.txt], path
file [/etc/elasticsearch/etc/elasticsearch/synonyms/location_synonyms.txt],
and classpath
Looks like the term synonyms file was blocking shards from being allocated.
Even though the files were in the listed locations (note they are relative
to the conf dir) with all permissions verified, I ended up having to dump
them in the classpath for ES to finally pick them up. That was the trick.
Everything is back to normal and recovering.
Thanks!
Andreas
On Tuesday, November 19, 2013 2:11:58 PM UTC-8, Andreas Echavez wrote:
Unfortunately I don't have time based indices nor do we open/close them.
Interestingly enough (for the sake of testing) it refuses to let me close
an index because the primary isn't allocated
{"error":"RemoteTransportException[[elasticsearch-prod-5][inet[/10.156.208.43:9300]][indices/close]];
nested: IndexPrimaryShardNotAllocatedException[[test_index4] primary not
allocated post api]; ","status":409}
If anyone had any other ideas it'd be greatly appreciated. It really feels
like a disk full or out of memory problem, but writing to existing indexes
works fine, just not creating new ones or routing shards.
Thanks,
Andreas
On Tuesday, November 19, 2013 12:57:50 PM UTC-8, Kurt Hurtado wrote:
So (assuming) you have time-based indices, do you close them once they
are no longer being written to?
Elasticsearch Platform — Find real-time answers at scale | Elastic
On Tuesday, November 19, 2013 12:45:00 PM UTC-8, Andreas Echavez wrote:
Trying with 1 shard and 0 replicas seems to be the same strange behavior
curl -XPUT 'http://localhost:9200/test_index4/' -d '{
"settings" : {
"index" : {
"number_of_shards" : 1,
"number_of_replicas" : 0
}
}
}'
{"ok":true,"acknowledged":false}root@es4:/data/elasticsearch#
curl -s -XGET '
http://localhost:9200/_cluster/health?pretty=true&level=indices' |
grep test_index4 -A10
"test_index4" : {
"status" : "red",
"number_of_shards" : 1,
"number_of_replicas" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},
Thanks,
Andreas
On Tuesday, November 19, 2013 12:37:00 PM UTC-8, Kurt Hurtado wrote:
Hi Andreas,
You said you reduced and increase replicas... have you also
experimented with number of shards?
Best,
Kurt
On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote:
How many shards do you have on your node (i.e. no. of indices * no. of
shards * no. of replicas) ? I was seeing a similar issues when I created a
lot of indices with a high number of shards on a 2 node cluster. In that
case, it is probably because of limited file descriptors. But that's my
assumption and I am fairly new to this world.
Vaidik Kapoor
vaidikkapoor.info
On 20 November 2013 01:55, Andreas Echavez oceanp...@gmail.comwrote:
Hi,
I'm seeing some problems with a two-node cluster where all my shards
have gone Yellow, and new shards are refusing to allocate.
Both running 90.5 & Java 1.7.0_25
I've tried almost everything I can think of to get my existing
indexes passed initialization (routing API, reducing and increasing
replicas, restarting and clearing out nodes) but no luck. Any ideas on how
to troubleshoot this one?
curl -XPUT 'http://localhost:9200/test_index/'
{"ok":true,"acknowledged":false}
curl -s -XGET '
http://localhost:9200/_cluster/health?pretty=true&level=indices' |
grep test_index -A15
"test_index" : {
"status" : "red",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 3,
"unassigned_shards" : 7
},
Shards seem to be stuck in initializing...
Node (os) stats
https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt
Cluster health
https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt
https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.