Cluster red status

Elastic Stack Elasticsearch
1

Hi all
after running ES ver 90.2 for weeks, the status changed to red. and here is
my cluster health:
{
cluster_name: Meteorites
status: red
timed_out: false
number_of_nodes: 1
number_of_data_nodes: 1
active_primary_shards: 96
active_shards: 96
relocating_shards: 0
initializing_shards: 0
unassigned_shards: 114
}

and here is my cluster nodes status:
{
ok: true
cluster_name: Meteorites
nodes: {
QDVMZqGCS0Cd2GKic7PtWw: {
name: Meteorite01
transport_address: inet[/192.168.1.45:9300]
version: 0.90.2
http_address: inet[/192.168.1.45:9200]
attributes: {
master: true
}
}
}
}

and last lines of my logs:

    at

org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at
org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at
org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at
org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at
org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at
org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:812)
at
org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:129)
at
org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:66)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:338)
... 8 more
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
sending failed shard for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
received shard failed for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,751][WARN ][indices.cluster ] [Meteorite01]
[logstash-2013.07.27][2] failed to create shard
org.elasticsearch.index.shard.IndexShardCreationException:
[logstash-2013.07.27][2] failed to create shard
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:340)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:558)

thanks in advance

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hello Iman,

Try checking if you're hitting the limit on the number of open files. By
increasing the open files limit :slight_smile: The way you're doing that depends on how
you installed ES. But here's a nice tutorial about it:

I'm not sure if this is the cause, but just searching around this error: "
IOException[directory XXX exists and is a directory, but cannot be listed:
list() returned null", returns a lot of "too many open files" associations.

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Sat, Jul 27, 2013 at 10:32 AM, Iman Khosravi imannapoleon@gmail.comwrote:

Hi all
after running ES ver 90.2 for weeks, the status changed to red. and here
is my cluster health:
{
cluster_name: Meteorites
status: red
timed_out: false
number_of_nodes: 1
number_of_data_nodes: 1
active_primary_shards: 96
active_shards: 96
relocating_shards: 0
initializing_shards: 0
unassigned_shards: 114
}

and here is my cluster nodes status:
{
ok: true
cluster_name: Meteorites
nodes: {
QDVMZqGCS0Cd2GKic7PtWw: {
name: Meteorite01
transport_address: inet[/192.168.1.45:9300]
version: 0.90.2
http_address: inet[/192.168.1.45:9200]
attributes: {
master: true
}
}
}
}

and last lines of my logs:

    at

org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at
org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at
org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at
org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at
org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at
org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:812)
at
org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:129)
at
org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:66)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:338)
... 8 more
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
sending failed shard for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
received shard failed for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,751][WARN ][indices.cluster ] [Meteorite01]
[logstash-2013.07.27][2] failed to create shard
org.elasticsearch.index.shard.IndexShardCreationException:
[logstash-2013.07.27][2] failed to create shard
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:340)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:558)

thanks in advance

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.