Cluster red status

Iman_Khosravi · July 27, 2013, 7:32am

Hi all
after running ES ver 90.2 for weeks, the status changed to red. and here is
my cluster health:
{
cluster_name: Meteorites
status: red
timed_out: false
number_of_nodes: 1
number_of_data_nodes: 1
active_primary_shards: 96
active_shards: 96
relocating_shards: 0
initializing_shards: 0
unassigned_shards: 114
}

and here is my cluster nodes status:
{
ok: true
cluster_name: Meteorites
nodes: {
QDVMZqGCS0Cd2GKic7PtWw: {
name: Meteorite01
transport_address: inet[/192.168.1.45:9300]
version: 0.90.2
http_address: inet[/192.168.1.45:9200]
attributes: {
master: true
}
}
}
}

and last lines of my logs:

at

org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at
org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at
org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at
org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at
org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at
org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:812)
at
org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:129)
at
org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:66)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:338)
... 8 more
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
sending failed shard for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
received shard failed for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,751][WARN ][indices.cluster ] [Meteorite01]
[logstash-2013.07.27][2] failed to create shard
org.elasticsearch.index.shard.IndexShardCreationException:
[logstash-2013.07.27][2] failed to create shard
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:340)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:558)

thanks in advance

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

radu_gheorghe · July 30, 2013, 2:33pm

Hello Iman,

Try checking if you're hitting the limit on the number of open files. By
increasing the open files limit The way you're doing that depends on how
you installed ES. But here's a nice tutorial about it:

I'm not sure if this is the cause, but just searching around this error: "
IOException[directory XXX exists and is a directory, but cannot be listed:
list() returned null", returns a lot of "too many open files" associations.

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Sat, Jul 27, 2013 at 10:32 AM, Iman Khosravi imannapoleon@gmail.comwrote:

Hi all
after running ES ver 90.2 for weeks, the status changed to red. and here
is my cluster health:
{
cluster_name: Meteorites
status: red
timed_out: false
number_of_nodes: 1
number_of_data_nodes: 1
active_primary_shards: 96
active_shards: 96
relocating_shards: 0
initializing_shards: 0
unassigned_shards: 114
}

and here is my cluster nodes status:
{
ok: true
cluster_name: Meteorites
nodes: {
QDVMZqGCS0Cd2GKic7PtWw: {
name: Meteorite01
transport_address: inet[/192.168.1.45:9300]
version: 0.90.2
http_address: inet[/192.168.1.45:9200]
attributes: {
master: true
}
}
}
}

and last lines of my logs:
    at
org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at
org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at
org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at
org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at
org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at
org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:812)
at
org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:129)
at
org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:66)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:338)
... 8 more
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
sending failed shard for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
received shard failed for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,751][WARN ][indices.cluster ] [Meteorite01]
[logstash-2013.07.27][2] failed to create shard
org.elasticsearch.index.shard.IndexShardCreationException:
[logstash-2013.07.27][2] failed to create shard
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:340)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:558)

thanks in advance

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Topic		Replies	Views
Cluster health red Elasticsearch	4	444	July 6, 2017
Mysterious "red" cluster status has happened ~4x now Elasticsearch	1	301	July 6, 2017
Elastic Cluster Red - How to fix! Elasticsearch	6	7320	June 13, 2019
Why Cluster Status is Red, please help Elasticsearch	9	1686	July 5, 2017
Status Red for cluster Elasticsearch	8	504	October 17, 2018

Cluster red status

Best regards, Radu

Related topics

Best regards,
Radu