Hi all
after running ES ver 90.2 for weeks, the status changed to red. and here is
my cluster health:
{
cluster_name: Meteorites
status: red
timed_out: false
number_of_nodes: 1
number_of_data_nodes: 1
active_primary_shards: 96
active_shards: 96
relocating_shards: 0
initializing_shards: 0
unassigned_shards: 114
}
and here is my cluster nodes status:
{
ok: true
cluster_name: Meteorites
nodes: {
QDVMZqGCS0Cd2GKic7PtWw: {
name: Meteorite01
transport_address: inet[/192.168.1.45:9300]
version: 0.90.2
http_address: inet[/192.168.1.45:9200]
attributes: {
master: true
}
}
}
}
and last lines of my logs:
at
org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at
org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at
org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at
org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at
org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at
org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:812)
at
org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:129)
at
org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:66)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:338)
... 8 more
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
sending failed shard for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
received shard failed for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,751][WARN ][indices.cluster ] [Meteorite01]
[logstash-2013.07.27][2] failed to create shard
org.elasticsearch.index.shard.IndexShardCreationException:
[logstash-2013.07.27][2] failed to create shard
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:340)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:558)
thanks in advance
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Hello Iman,
Try checking if you're hitting the limit on the number of open files. By
increasing the open files limit 
The way you're doing that depends on how
you installed ES. But here's a nice tutorial about it:
http://www.elasticsearch.org/tutorials/too-many-open-files/
I'm not sure if this is the cause, but just searching around this error: "
IOException[directory XXX exists and is a directory, but cannot be listed:
list() returned null", returns a lot of "too many open files" associations.
Best regards,
Radu
http://sematext.com/ -- ElasticSearch -- Solr -- Lucene
On Sat, Jul 27, 2013 at 10:32 AM, Iman Khosravi imannapoleon@gmail.comwrote:
Hi all
after running ES ver 90.2 for weeks, the status changed to red. and here
is my cluster health:
{
cluster_name: Meteorites
status: red
timed_out: false
number_of_nodes: 1
number_of_data_nodes: 1
active_primary_shards: 96
active_shards: 96
relocating_shards: 0
initializing_shards: 0
unassigned_shards: 114
}
and here is my cluster nodes status:
{
ok: true
cluster_name: Meteorites
nodes: {
QDVMZqGCS0Cd2GKic7PtWw: {
name: Meteorite01
transport_address: inet[/192.168.1.45:9300]
version: 0.90.2
http_address: inet[/192.168.1.45:9200]
attributes: {
master: true
}
}
}
}
and last lines of my logs:
at
org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at
org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at
org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at
org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at
org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at
org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at
org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at
org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:812)
at
org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at
org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:129)
at
org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:66)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:338)
... 8 more
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
sending failed shard for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,730][WARN ][cluster.action.shard ] [Meteorite01]
received shard failed for [logstash-2013.07.27][2],
node[TgU-CZplT1-tfaJlqpfryg], [P], s[INITIALIZING], reason [Failed to
create shard, message [IndexShardCreationException[[logstash-2013.07.27][2]
failed to create shard]; nested: IOException[directory
'/elastic/Meteorites/nodes/0/indices/logstash-2013.07.27/2/index' exists
and is a directory, but cannot be listed: list() returned null]; ]]
[2013-07-27 09:13:44,751][WARN ][indices.cluster ] [Meteorite01]
[logstash-2013.07.27][2] failed to create shard
org.elasticsearch.index.shard.IndexShardCreationException:
[logstash-2013.07.27][2] failed to create shard
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:340)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:558)
thanks in advance
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.