River flows for a while, then stops with exception

Elastic Stack Elasticsearch
1

Hey guys,

I've got a RabbitMQ river that runs for a while, gets through a few
million entries, and then stops running. On my side, when I try to do:

curl -XGET localhost:9200/_river/logstash/_meta

I get:

{"error":"NoShardAvailableActionException[[_river][0] No shard
available for [[_river][logstash][_meta]: routing
[null]]]","status":500}

This seems to happen any time I run the river for a long while, and
the only fix is to erase the index completely.

Any ideas?

Thanks,
Greg Rice

2

Here's a bit more:

[2012-05-18 03:43:42,879][WARN ][cluster.action.shard ]
[Evilhawk<200e>] sending failed shard for [logstash-2012.05.17][3],
node[TdEOVnA8QRqH8pW1p39bhA],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [logstash-2012.05.17][3]: Recovery
failed from [Dorma][2NCYRR
DkNToK4JyfQn4trIA][inet[/10.170.110.130:9300]] into
[Evilhawk<200e>][TdEOVnA8QRqH8pW1p39bhA][inet[/10.170.110.119:9300]]];
nested: RemoteTransportException[[DD
orma][inet[/10.170.110.130:9300]][index/shard/recovery/startRecovery]];
nested: RecoveryEngineException[[logstash-2012.05.17][3] Phase[1] Execution
failed]; nn
ested: RecoverFilesRecoveryException[[logstash-2012.05.17][3] Failed to
transfer [153] files with total size of [850.2mb]]; nested:
FileNotFoundException[/datt
a/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/_6ui.nrm (Too
many open files)]; ]]
[2012-05-18 03:43:55,098][INFO ][node ]
[Evilhawk<200e>] {0.19.3}[14925]: stopping ...
[2012-05-18 03:43:55,104][INFO ][river.rabbitmq ]
[Evilhawk<200e>] [rabbitmq][logstash-0_0_0_0] closing rabbitmq river
[2012-05-18 03:44:16,928][INFO ][node ] [Terraxia]
{0.19.3}[19822]: initializing ...
[2012-05-18 03:44:16,938][INFO ][plugins ] [Terraxia]
loaded [river-rabbitmq], sites
[2012-05-18 03:44:20,420][INFO ][node ] [Terraxia]
{0.19.3}[19822]: initialized
[2012-05-18 03:44:20,420][INFO ][node ] [Terraxia]
{0.19.3}[19822]: starting ...
[2012-05-18 03:44:21,855][INFO ][transport ] [Terraxia]
bound_address {inet[/0.0.0.0:9301]}, publish_address
{inet[/10.170.110.119:9301]}
[2012-05-18 03:44:25,014][INFO ][cluster.service ] [Terraxia]
detected_master [Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]], aa
dded
{[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason:
zen-disco--
receive(from master [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 03:44:29,454][INFO ][discovery ] [Terraxia]
elasticsearch/2-Co3TxbRlqXHIhd597D_A
[2012-05-18 03:44:29,460][INFO ][http ] [Terraxia]
bound_address {inet[/0.0.0.0:9201]}, publish_address
{inet[/10.170.110.119:9201]}
[2012-05-18 03:44:29,460][INFO ][node ] [Terraxia]
{0.19.3}[19822]: started
[2012-05-18 03:44:35,864][INFO ][node ]
[Evilhawk<200e>] {0.19.3}[14925]: stopped
[2012-05-18 03:44:35,864][INFO ][node ]
[Evilhawk<200e>] {0.19.3}[14925]: closing ...
[2012-05-18 03:44:45,878][INFO ][node ]
[Evilhawk<200e>] {0.19.3}[14925]: closed
[2012-05-18 03:45:00,834][INFO ][cluster.service ] [Terraxia]
added {[Psi-Lord][kJv_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],},
reason: zen-dii
sco-receive(from master [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 03:47:35,093][INFO ][cluster.service ] [Terraxia]
removed {[Psi-Lord][kJv_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],},
reason: zen--
disco-receive(from master [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 22:14:18,472][INFO ][discovery.zen ] [Terraxia]
master_left [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reaa
son [shut_down]
[2012-05-18 22:14:18,500][INFO ][discovery.zen ] [Terraxia]
master_left [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reaa
son [transport disconnected (with verified connect)]
[2012-05-18 22:14:18,500][INFO ][cluster.service ] [Terraxia]
master {new [Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],
previous [Nee
well, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]}, removed
{[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],},
reason: zz
en-disco-master_failed ([Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]])
[2012-05-18 22:14:54,233][INFO ][cluster.service ] [Terraxia]
added {[Loss][FXpkOT72Rfii02DRD6BPSQ][inet[/10.170.110.118:9300]],},
reason: zen-disco--
receive(from master
[[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]]])

                                                            119,1

Bot
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by:
org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
[logstash-2012.05.17][3] Failed to transfer [153] files with total size of
[850.2mb]
at
org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:188)
at
org.elasticsearch.index.engine.robin.RobinEngine.recover(RobinEngine.java:1058)
... 9 more
Caused by: java.io.FileNotFoundException:
/data/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/_6ui.nrm
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:433)
at
org.elasticsearch.indices.recovery.RecoverySource$1$1.run(RecoverySource.java:137)
... 3 more
[2012-05-18 03:43:42,879][WARN ][cluster.action.shard ]
[Evilhawk<200e>] sending failed shard for [logstash-2012.05.17][3],
node[TdEOVnA8QRqH8pW1p39bhA], [R], s[INITIALIZING], reason [Failed to start
shard, message [RecoveryFailedException[Index Shard
[logstash-2012.05.17][3]: Recovery failed from
[Dorma][2NCYRDkNToK4JyfQn4trIA][inet[/10.170.110.130:9300]] into
[Evilhawk<200e>][TdEOVnA8QRqH8pW1p39bhA][inet[/10.170.110.119:9300]]];
nested:
RemoteTransportException[[Dorma][inet[/10.170.110.130:9300]][index/shard/recovery/startRecovery]];
nested: RecoveryEngineException[[logstash-2012.05.17][3] Phase[1] Execution
failed]; nested: RecoverFilesRecoveryException[[logstash-2012.05.17][3]
Failed to transfer [153] files with total size of [850.2mb]]; nested:
FileNotFoundException[/data/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/_6ui.nrm
(Too many open files)]; ]]
[2012-05-18 03:43:55,098][INFO ][node ]
[Evilhawk<200e>] {0.19.3}[14925]: stopping ...
[2012-05-18 03:43:55,104][INFO ][river.rabbitmq ]
[Evilhawk<200e>] [rabbitmq][logstash-0_0_0_0] closing rabbitmq river
[2012-05-18 03:44:16,928][INFO ][node ] [Terraxia]
{0.19.3}[19822]: initializing ...
[2012-05-18 03:44:16,938][INFO ][plugins ] [Terraxia]
loaded [river-rabbitmq], sites
[2012-05-18 03:44:20,420][INFO ][node ] [Terraxia]
{0.19.3}[19822]: initialized
[2012-05-18 03:44:20,420][INFO ][node ] [Terraxia]
{0.19.3}[19822]: starting ...
[2012-05-18 03:44:21,855][INFO ][transport ] [Terraxia]
bound_address {inet[/0.0.0.0:9301]}, publish_address
{inet[/10.170.110.119:9301]}
[2012-05-18 03:44:25,014][INFO ][cluster.service ] [Terraxia]
detected_master [Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]], added
{[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason:
zen-disco-receive(from master [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 03:44:29,454][INFO ][discovery ] [Terraxia]
elasticsearch/2-Co3TxbRlqXHIhd597D_A
[2012-05-18 03:44:29,460][INFO ][http ] [Terraxia]
bound_address {inet[/0.0.0.0:9201]}, publish_address
{inet[/10.170.110.119:9201]}
[2012-05-18 03:44:29,460][INFO ][node ] [Terraxia]
{0.19.3}[19822]: started
[2012-05-18 03:44:35,864][INFO ][node ]
[Evilhawk<200e>] {0.19.3}[14925]: stopped
[2012-05-18 03:44:35,864][INFO ][node ]
[Evilhawk<200e>] {0.19.3}[14925]: closing ...
[2012-05-18 03:44:45,878][INFO ][node ]
[Evilhawk<200e>] {0.19.3}[14925]: closed
[2012-05-18 03:45:00,834][INFO ][cluster.service ] [Terraxia]
added {[Psi-Lord][kJv_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],},
reason: zen-disco-receive(from master [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 03:47:35,093][INFO ][cluster.service ] [Terraxia]
removed {[Psi-Lord][kJv_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],},
reason: zen-disco-receive(from master [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 22:14:18,472][INFO ][discovery.zen ] [Terraxia]
master_left [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reason
[shut_down]
[2012-05-18 22:14:18,500][INFO ][discovery.zen ] [Terraxia]
master_left [[Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reason
[transport disconnected (with verified connect)]
[2012-05-18 22:14:18,500][INFO ][cluster.service ] [Terraxia]
master {new [Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],
previous [Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]}, removed
{[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],},
reason: zen-disco-master_failed ([Newell,
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]])
[2012-05-18 22:14:54,233][INFO ][cluster.service ] [Terraxia]
added {[Loss][FXpkOT72Rfii02DRD6BPSQ][inet[/10.170.110.118:9300]],},
reason: zen-disco-receive(from master
[[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]]])

3

Too many open files is a "common" error.
See : Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH
David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 19 mai 2012 à 00:53, Gregory Rice gregrice@gmail.com a écrit :

Here's a bit more:

[2012-05-18 03:43:42,879][WARN ][cluster.action.shard ] [Evilhawk<200e>] sending failed shard for [logstash-2012.05.17][3], node[TdEOVnA8QRqH8pW1p39bhA],
[R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [logstash-2012.05.17][3]: Recovery failed from [Dorma][2NCYRR
DkNToK4JyfQn4trIA][inet[/10.170.110.130:9300]] into [Evilhawk<200e>][TdEOVnA8QRqH8pW1p39bhA][inet[/10.170.110.119:9300]]]; nested: RemoteTransportException[[DD
orma][inet[/10.170.110.130:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[logstash-2012.05.17][3] Phase[1] Execution failed]; nn
ested: RecoverFilesRecoveryException[[logstash-2012.05.17][3] Failed to transfer [153] files with total size of [850.2mb]]; nested: FileNotFoundException[/datt
a/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/_6ui.nrm (Too many open files)]; ]]
[2012-05-18 03:43:55,098][INFO ][node ] [Evilhawk<200e>] {0.19.3}[14925]: stopping ...
[2012-05-18 03:43:55,104][INFO ][river.rabbitmq ] [Evilhawk<200e>] [rabbitmq][logstash-0_0_0_0] closing rabbitmq river
[2012-05-18 03:44:16,928][INFO ][node ] [Terraxia] {0.19.3}[19822]: initializing ...
[2012-05-18 03:44:16,938][INFO ][plugins ] [Terraxia] loaded [river-rabbitmq], sites
[2012-05-18 03:44:20,420][INFO ][node ] [Terraxia] {0.19.3}[19822]: initialized
[2012-05-18 03:44:20,420][INFO ][node ] [Terraxia] {0.19.3}[19822]: starting ...
[2012-05-18 03:44:21,855][INFO ][transport ] [Terraxia] bound_address {inet[/0.0.0.0:9301]}, publish_address {inet[/10.170.110.119:9301]}
[2012-05-18 03:44:25,014][INFO ][cluster.service ] [Terraxia] detected_master [Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]], aa
dded {[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason: zen-disco--
receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 03:44:29,454][INFO ][discovery ] [Terraxia] elasticsearch/2-Co3TxbRlqXHIhd597D_A
[2012-05-18 03:44:29,460][INFO ][http ] [Terraxia] bound_address {inet[/0.0.0.0:9201]}, publish_address {inet[/10.170.110.119:9201]}
[2012-05-18 03:44:29,460][INFO ][node ] [Terraxia] {0.19.3}[19822]: started
[2012-05-18 03:44:35,864][INFO ][node ] [Evilhawk<200e>] {0.19.3}[14925]: stopped
[2012-05-18 03:44:35,864][INFO ][node ] [Evilhawk<200e>] {0.19.3}[14925]: closing ...
[2012-05-18 03:44:45,878][INFO ][node ] [Evilhawk<200e>] {0.19.3}[14925]: closed
[2012-05-18 03:45:00,834][INFO ][cluster.service ] [Terraxia] added {[Psi-Lord][kJv_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],}, reason: zen-dii
sco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 03:47:35,093][INFO ][cluster.service ] [Terraxia] removed {[Psi-Lord][kJv_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],}, reason: zen--
disco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 22:14:18,472][INFO ][discovery.zen ] [Terraxia] master_left [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reaa
son [shut_down]
[2012-05-18 22:14:18,500][INFO ][discovery.zen ] [Terraxia] master_left [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reaa
son [transport disconnected (with verified connect)]
[2012-05-18 22:14:18,500][INFO ][cluster.service ] [Terraxia] master {new [Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]], previous [Nee
well, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]}, removed {[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason: zz
en-disco-master_failed ([Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]])
[2012-05-18 22:14:54,233][INFO ][cluster.service ] [Terraxia] added {[Loss][FXpkOT72Rfii02DRD6BPSQ][inet[/10.170.110.118:9300]],}, reason: zen-disco--
receive(from master [[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]]])
119,1 Bot
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [logstash-2012.05.17][3] Failed to transfer [153] files with total size of [850.2mb]
at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:188)
at org.elasticsearch.index.engine.robin.RobinEngine.recover(RobinEngine.java:1058)
... 9 more
Caused by: java.io.FileNotFoundException: /data/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/_6ui.nrm (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)
at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:433)
at org.elasticsearch.indices.recovery.RecoverySource$1$1.run(RecoverySource.java:137)
... 3 more
[2012-05-18 03:43:42,879][WARN ][cluster.action.shard ] [Evilhawk<200e>] sending failed shard for [logstash-2012.05.17][3], node[TdEOVnA8QRqH8pW1p39bhA], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [logstash-2012.05.17][3]: Recovery failed from [Dorma][2NCYRDkNToK4JyfQn4trIA][inet[/10.170.110.130:9300]] into [Evilhawk<200e>][TdEOVnA8QRqH8pW1p39bhA][inet[/10.170.110.119:9300]]]; nested: RemoteTransportException[[Dorma][inet[/10.170.110.130:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[logstash-2012.05.17][3] Phase[1] Execution failed]; nested: RecoverFilesRecoveryException[[logstash-2012.05.17][3] Failed to transfer [153] files with total size of [850.2mb]]; nested: FileNotFoundException[/data/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/_6ui.nrm (Too many open files)]; ]]
[2012-05-18 03:43:55,098][INFO ][node ] [Evilhawk<200e>] {0.19.3}[14925]: stopping ...
[2012-05-18 03:43:55,104][INFO ][river.rabbitmq ] [Evilhawk<200e>] [rabbitmq][logstash-0_0_0_0] closing rabbitmq river
[2012-05-18 03:44:16,928][INFO ][node ] [Terraxia] {0.19.3}[19822]: initializing ...
[2012-05-18 03:44:16,938][INFO ][plugins ] [Terraxia] loaded [river-rabbitmq], sites
[2012-05-18 03:44:20,420][INFO ][node ] [Terraxia] {0.19.3}[19822]: initialized
[2012-05-18 03:44:20,420][INFO ][node ] [Terraxia] {0.19.3}[19822]: starting ...
[2012-05-18 03:44:21,855][INFO ][transport ] [Terraxia] bound_address {inet[/0.0.0.0:9301]}, publish_address {inet[/10.170.110.119:9301]}
[2012-05-18 03:44:25,014][INFO ][cluster.service ] [Terraxia] detected_master [Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]], added {[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason: zen-disco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 03:44:29,454][INFO ][discovery ] [Terraxia] elasticsearch/2-Co3TxbRlqXHIhd597D_A
[2012-05-18 03:44:29,460][INFO ][http ] [Terraxia] bound_address {inet[/0.0.0.0:9201]}, publish_address {inet[/10.170.110.119:9201]}
[2012-05-18 03:44:29,460][INFO ][node ] [Terraxia] {0.19.3}[19822]: started
[2012-05-18 03:44:35,864][INFO ][node ] [Evilhawk<200e>] {0.19.3}[14925]: stopped
[2012-05-18 03:44:35,864][INFO ][node ] [Evilhawk<200e>] {0.19.3}[14925]: closing ...
[2012-05-18 03:44:45,878][INFO ][node ] [Evilhawk<200e>] {0.19.3}[14925]: closed
[2012-05-18 03:45:00,834][INFO ][cluster.service ] [Terraxia] added {[Psi-Lord][kJv_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],}, reason: zen-disco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 03:47:35,093][INFO ][cluster.service ] [Terraxia] removed {[Psi-Lord][kJv_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],}, reason: zen-disco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])
[2012-05-18 22:14:18,472][INFO ][discovery.zen ] [Terraxia] master_left [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reason [shut_down]
[2012-05-18 22:14:18,500][INFO ][discovery.zen ] [Terraxia] master_left [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reason [transport disconnected (with verified connect)]
[2012-05-18 22:14:18,500][INFO ][cluster.service ] [Terraxia] master {new [Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]], previous [Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]}, removed {[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason: zen-disco-master_failed ([Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]])
[2012-05-18 22:14:54,233][INFO ][cluster.service ] [Terraxia] added {[Loss][FXpkOT72Rfii02DRD6BPSQ][inet[/10.170.110.118:9300]],}, reason: zen-disco-receive(from master [[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]]])

4

Hi dadoonet, that link seems to be dead (got a 404).

Can you please point to a reference or explain what do you mean with «too many open files is a "common" error» ?

Thanks!
Pedro