# River flows for a while, then stops with exception

**URL:** <https://discuss.elastic.co/t/river-flows-for-a-while-then-stops-with-exception/7763>\
**Category:** Elasticsearch\
**Created:** [May 18, 2012, 4:35pm UTC](https://discuss.elastic.co/t/river-flows-for-a-while-then-stops-with-exception/7763 "2012-05-18T16:35:18Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gregory\_Rice](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregory_rice/32/2884_2.png) [@Gregory\_Rice](https://discuss.elastic.co/u/Gregory_Rice)\
**Post date:** [May 18, 2012, 4:35pm UTC](https://discuss.elastic.co/t/river-flows-for-a-while-then-stops-with-exception/7763/1 "2012-05-18T16:35:18Z")

</div>

Hey guys,

I've got a RabbitMQ river that runs for a while, gets through a few  
million entries, and then stops running. On my side, when I try to do:

curl -XGET localhost:9200/\_river/logstash/\_meta

I get:

{"error":"NoShardAvailableActionException[[\_river][0] No shard  
available for [[\_river][logstash][\_meta]: routing  
[null]]]","status":500}

This seems to happen any time I run the river for a long while, and  
the only fix is to erase the index completely.

Any ideas?

Thanks,  
Greg Rice

---

<div class="post-metadata">

**Author:** ![Gregory\_Rice](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregory_rice/32/2884_2.png) [@Gregory\_Rice](https://discuss.elastic.co/u/Gregory_Rice)\
**Post date:** [May 18, 2012, 10:53pm UTC](https://discuss.elastic.co/t/river-flows-for-a-while-then-stops-with-exception/7763/2 "2012-05-18T22:53:49Z")

</div>

> Here's a bit more:

[2012-05-18 03:43:42,879][WARN][cluster.action.shard]  
[Evilhawk\<200e\>] sending failed shard for [logstash-2012.05.17][3],  
node[TdEOVnA8QRqH8pW1p39bhA],  
[R], s[INITIALIZING], reason [Failed to start shard, message  
[RecoveryFailedException[Index Shard [logstash-2012.05.17][3]: Recovery  
failed from [Dorma][2NCYRR  
DkNToK4JyfQn4trIA][inet[/10.170.110.130:9300]] into  
[Evilhawk\<200e\>][TdEOVnA8QRqH8pW1p39bhA][inet[/10.170.110.119:9300]]];  
nested: RemoteTransportException[[DD  
orma][inet[/10.170.110.130:9300]][index/shard/recovery/startRecovery]];  
nested: RecoveryEngineException[[logstash-2012.05.17][3] Phase[1] Execution  
failed]; nn  
ested: RecoverFilesRecoveryException[[logstash-2012.05.17][3] Failed to  
transfer [153] files with total size of [850.2mb]]; nested:  
FileNotFoundException[/datt  
a/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/\_6ui.nrm (Too  
many open files)]; ]]  
[2012-05-18 03:43:55,098][INFO][node]  
[Evilhawk\<200e\>] {0.19.3}[14925]: stopping ...  
[2012-05-18 03:43:55,104][INFO][river.rabbitmq]  
[Evilhawk\<200e\>] [rabbitmq][logstash-0\_0\_0\_0] closing rabbitmq river  
[2012-05-18 03:44:16,928][INFO][node] [Terraxia]  
{0.19.3}[19822]: initializing ...  
[2012-05-18 03:44:16,938][INFO][plugins] [Terraxia]  
loaded [river-rabbitmq], sites   
[2012-05-18 03:44:20,420][INFO][node] [Terraxia]  
{0.19.3}[19822]: initialized  
[2012-05-18 03:44:20,420][INFO][node] [Terraxia]  
{0.19.3}[19822]: starting ...  
[2012-05-18 03:44:21,855][INFO][transport] [Terraxia]  
bound\_address {inet[/0.0.0.0:9301]}, publish\_address  
{inet[/10.170.110.119:9301]}  
[2012-05-18 03:44:25,014][INFO][cluster.service] [Terraxia]  
detected\_master [Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]], aa  
dded  
{[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason:  
zen-disco--  
receive(from master [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
[2012-05-18 03:44:29,454][INFO][discovery] [Terraxia]  
elasticsearch/2-Co3TxbRlqXHIhd597D\_A  
[2012-05-18 03:44:29,460][INFO][http] [Terraxia]  
bound\_address {inet[/0.0.0.0:9201]}, publish\_address  
{inet[/10.170.110.119:9201]}  
[2012-05-18 03:44:29,460][INFO][node] [Terraxia]  
{0.19.3}[19822]: started  
[2012-05-18 03:44:35,864][INFO][node]  
[Evilhawk\<200e\>] {0.19.3}[14925]: stopped  
[2012-05-18 03:44:35,864][INFO][node]  
[Evilhawk\<200e\>] {0.19.3}[14925]: closing ...  
[2012-05-18 03:44:45,878][INFO][node]  
[Evilhawk\<200e\>] {0.19.3}[14925]: closed  
[2012-05-18 03:45:00,834][INFO][cluster.service] [Terraxia]  
added {[Psi-Lord][kJv\_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],},  
reason: zen-dii  
sco-receive(from master [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
[2012-05-18 03:47:35,093][INFO][cluster.service] [Terraxia]  
removed {[Psi-Lord][kJv\_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],},  
reason: zen--  
disco-receive(from master [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
[2012-05-18 22:14:18,472][INFO][discovery.zen] [Terraxia]  
master\_left [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reaa  
son [shut\_down]  
[2012-05-18 22:14:18,500][INFO][discovery.zen] [Terraxia]  
master\_left [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reaa  
son [transport disconnected (with verified connect)]  
[2012-05-18 22:14:18,500][INFO][cluster.service] [Terraxia]  
master {new [Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],  
previous [Nee  
well, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]}, removed  
{[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],},  
reason: zz  
en-disco-master\_failed ([Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]])  
[2012-05-18 22:14:54,233][INFO][cluster.service] [Terraxia]  
added {[Loss][FXpkOT72Rfii02DRD6BPSQ][inet[/10.170.110.118:9300]],},  
reason: zen-disco--  
receive(from master  
[[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]]])

```
                                                            119,1       

```

Bot  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
at java.lang.Thread.run(Thread.java:636)  
Caused by:  
org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:  
[logstash-2012.05.17][3] Failed to transfer [153] files with total size of  
[850.2mb]  
at  
org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:188)  
at  
org.elasticsearch.index.engine.robin.RobinEngine.recover(RobinEngine.java:1058)  
... 9 more  
Caused by: java.io.FileNotFoundException:  
/data/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/\_6ui.nrm  
(Too many open files)  
at java.io.RandomAccessFile.open(Native Method)  
at java.io.RandomAccessFile.(RandomAccessFile.java:233)  
at  
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)  
at  
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)  
at  
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)  
at  
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)  
at  
org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)  
at  
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:433)  
at  
org.elasticsearch.indices.recovery.RecoverySource$1$1.run(RecoverySource.java:137)  
... 3 more  
[2012-05-18 03:43:42,879][WARN][cluster.action.shard]  
[Evilhawk\<200e\>] sending failed shard for [logstash-2012.05.17][3],  
node[TdEOVnA8QRqH8pW1p39bhA], [R], s[INITIALIZING], reason [Failed to start  
shard, message [RecoveryFailedException[Index Shard  
[logstash-2012.05.17][3]: Recovery failed from  
[Dorma][2NCYRDkNToK4JyfQn4trIA][inet[/10.170.110.130:9300]] into  
[Evilhawk\<200e\>][TdEOVnA8QRqH8pW1p39bhA][inet[/10.170.110.119:9300]]];  
nested:  
RemoteTransportException[[Dorma][inet[/10.170.110.130:9300]][index/shard/recovery/startRecovery]];  
nested: RecoveryEngineException[[logstash-2012.05.17][3] Phase[1] Execution  
failed]; nested: RecoverFilesRecoveryException[[logstash-2012.05.17][3]  
Failed to transfer [153] files with total size of [850.2mb]]; nested:  
FileNotFoundException[/data/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/\_6ui.nrm  
(Too many open files)]; ]]  
[2012-05-18 03:43:55,098][INFO][node]  
[Evilhawk\<200e\>] {0.19.3}[14925]: stopping ...  
[2012-05-18 03:43:55,104][INFO][river.rabbitmq]  
[Evilhawk\<200e\>] [rabbitmq][logstash-0\_0\_0\_0] closing rabbitmq river  
[2012-05-18 03:44:16,928][INFO][node] [Terraxia]  
{0.19.3}[19822]: initializing ...  
[2012-05-18 03:44:16,938][INFO][plugins] [Terraxia]  
loaded [river-rabbitmq], sites   
[2012-05-18 03:44:20,420][INFO][node] [Terraxia]  
{0.19.3}[19822]: initialized  
[2012-05-18 03:44:20,420][INFO][node] [Terraxia]  
{0.19.3}[19822]: starting ...  
[2012-05-18 03:44:21,855][INFO][transport] [Terraxia]  
bound\_address {inet[/0.0.0.0:9301]}, publish\_address  
{inet[/10.170.110.119:9301]}  
[2012-05-18 03:44:25,014][INFO][cluster.service] [Terraxia]  
detected\_master [Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]], added  
{[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason:  
zen-disco-receive(from master [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
[2012-05-18 03:44:29,454][INFO][discovery] [Terraxia]  
elasticsearch/2-Co3TxbRlqXHIhd597D\_A  
[2012-05-18 03:44:29,460][INFO][http] [Terraxia]  
bound\_address {inet[/0.0.0.0:9201]}, publish\_address  
{inet[/10.170.110.119:9201]}  
[2012-05-18 03:44:29,460][INFO][node] [Terraxia]  
{0.19.3}[19822]: started  
[2012-05-18 03:44:35,864][INFO][node]  
[Evilhawk\<200e\>] {0.19.3}[14925]: stopped  
[2012-05-18 03:44:35,864][INFO][node]  
[Evilhawk\<200e\>] {0.19.3}[14925]: closing ...  
[2012-05-18 03:44:45,878][INFO][node]  
[Evilhawk\<200e\>] {0.19.3}[14925]: closed  
[2012-05-18 03:45:00,834][INFO][cluster.service] [Terraxia]  
added {[Psi-Lord][kJv\_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],},  
reason: zen-disco-receive(from master [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
[2012-05-18 03:47:35,093][INFO][cluster.service] [Terraxia]  
removed {[Psi-Lord][kJv\_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],},  
reason: zen-disco-receive(from master [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
[2012-05-18 22:14:18,472][INFO][discovery.zen] [Terraxia]  
master\_left [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reason  
[shut\_down]  
[2012-05-18 22:14:18,500][INFO][discovery.zen] [Terraxia]  
master\_left [[Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reason  
[transport disconnected (with verified connect)]  
[2012-05-18 22:14:18,500][INFO][cluster.service] [Terraxia]  
master {new [Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],  
previous [Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]}, removed  
{[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],},  
reason: zen-disco-master\_failed ([Newell,  
Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]])  
[2012-05-18 22:14:54,233][INFO][cluster.service] [Terraxia]  
added {[Loss][FXpkOT72Rfii02DRD6BPSQ][inet[/10.170.110.118:9300]],},  
reason: zen-disco-receive(from master  
[[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]]])

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 19, 2012, 12:47am UTC](https://discuss.elastic.co/t/river-flows-for-a-while-then-stops-with-exception/7763/3 "2012-05-19T00:47:43Z")

</div>

Too many open files is a "common" error.  
See : [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/i/nstallation.html)

HTH  
David 😉  
Twitter : @dadoonet / @elasticsearchfr

Le 19 mai 2012 à 00:53, Gregory Rice [gregrice@gmail.com](mailto:gregrice@gmail.com) a écrit :

> Here's a bit more:
> 
> [2012-05-18 03:43:42,879][WARN][cluster.action.shard] [Evilhawk\<200e\>] sending failed shard for [logstash-2012.05.17][3], node[TdEOVnA8QRqH8pW1p39bhA],  
> [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [logstash-2012.05.17][3]: Recovery failed from [Dorma][2NCYRR  
> DkNToK4JyfQn4trIA][inet[/10.170.110.130:9300]] into [Evilhawk\<200e\>][TdEOVnA8QRqH8pW1p39bhA][inet[/10.170.110.119:9300]]]; nested: RemoteTransportException[[DD  
> orma][inet[/10.170.110.130:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[logstash-2012.05.17][3] Phase[1] Execution failed]; nn  
> ested: RecoverFilesRecoveryException[[logstash-2012.05.17][3] Failed to transfer [153] files with total size of [850.2mb]]; nested: FileNotFoundException[/datt  
> a/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/\_6ui.nrm (Too many open files)]; ]]  
> [2012-05-18 03:43:55,098][INFO][node] [Evilhawk\<200e\>] {0.19.3}[14925]: stopping ...  
> [2012-05-18 03:43:55,104][INFO][river.rabbitmq] [Evilhawk\<200e\>] [rabbitmq][logstash-0\_0\_0\_0] closing rabbitmq river  
> [2012-05-18 03:44:16,928][INFO][node] [Terraxia] {0.19.3}[19822]: initializing ...  
> [2012-05-18 03:44:16,938][INFO][plugins] [Terraxia] loaded [river-rabbitmq], sites   
> [2012-05-18 03:44:20,420][INFO][node] [Terraxia] {0.19.3}[19822]: initialized  
> [2012-05-18 03:44:20,420][INFO][node] [Terraxia] {0.19.3}[19822]: starting ...  
> [2012-05-18 03:44:21,855][INFO][transport] [Terraxia] bound\_address {inet[/0.0.0.0:9301]}, publish\_address {inet[/10.170.110.119:9301]}  
> [2012-05-18 03:44:25,014][INFO][cluster.service] [Terraxia] detected\_master [Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]], aa  
> dded {[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason: zen-disco--  
> receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
> [2012-05-18 03:44:29,454][INFO][discovery] [Terraxia] elasticsearch/2-Co3TxbRlqXHIhd597D\_A  
> [2012-05-18 03:44:29,460][INFO][http] [Terraxia] bound\_address {inet[/0.0.0.0:9201]}, publish\_address {inet[/10.170.110.119:9201]}  
> [2012-05-18 03:44:29,460][INFO][node] [Terraxia] {0.19.3}[19822]: started  
> [2012-05-18 03:44:35,864][INFO][node] [Evilhawk\<200e\>] {0.19.3}[14925]: stopped  
> [2012-05-18 03:44:35,864][INFO][node] [Evilhawk\<200e\>] {0.19.3}[14925]: closing ...  
> [2012-05-18 03:44:45,878][INFO][node] [Evilhawk\<200e\>] {0.19.3}[14925]: closed  
> [2012-05-18 03:45:00,834][INFO][cluster.service] [Terraxia] added {[Psi-Lord][kJv\_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],}, reason: zen-dii  
> sco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
> [2012-05-18 03:47:35,093][INFO][cluster.service] [Terraxia] removed {[Psi-Lord][kJv\_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],}, reason: zen--  
> disco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
> [2012-05-18 22:14:18,472][INFO][discovery.zen] [Terraxia] master\_left [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reaa  
> son [shut\_down]  
> [2012-05-18 22:14:18,500][INFO][discovery.zen] [Terraxia] master\_left [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reaa  
> son [transport disconnected (with verified connect)]  
> [2012-05-18 22:14:18,500][INFO][cluster.service] [Terraxia] master {new [Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]], previous [Nee  
> well, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]}, removed {[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason: zz  
> en-disco-master\_failed ([Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]])  
> [2012-05-18 22:14:54,233][INFO][cluster.service] [Terraxia] added {[Loss][FXpkOT72Rfii02DRD6BPSQ][inet[/10.170.110.118:9300]],}, reason: zen-disco--  
> receive(from master [[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]]])  
> 119,1 Bot  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
> at java.lang.Thread.run(Thread.java:636)  
> Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [logstash-2012.05.17][3] Failed to transfer [153] files with total size of [850.2mb]  
> at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:188)  
> at org.elasticsearch.index.engine.robin.RobinEngine.recover(RobinEngine.java:1058)  
> ... 9 more  
> Caused by: java.io.FileNotFoundException: /data/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/\_6ui.nrm (Too many open files)  
> at java.io.RandomAccessFile.open(Native Method)  
> at java.io.RandomAccessFile.(RandomAccessFile.java:233)  
> at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)  
> at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)  
> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)  
> at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)  
> at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)  
> at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:433)  
> at org.elasticsearch.indices.recovery.RecoverySource$1$1.run(RecoverySource.java:137)  
> ... 3 more  
> [2012-05-18 03:43:42,879][WARN][cluster.action.shard] [Evilhawk\<200e\>] sending failed shard for [logstash-2012.05.17][3], node[TdEOVnA8QRqH8pW1p39bhA], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [logstash-2012.05.17][3]: Recovery failed from [Dorma][2NCYRDkNToK4JyfQn4trIA][inet[/10.170.110.130:9300]] into [Evilhawk\<200e\>][TdEOVnA8QRqH8pW1p39bhA][inet[/10.170.110.119:9300]]]; nested: RemoteTransportException[[Dorma][inet[/10.170.110.130:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[logstash-2012.05.17][3] Phase[1] Execution failed]; nested: RecoverFilesRecoveryException[[logstash-2012.05.17][3] Failed to transfer [153] files with total size of [850.2mb]]; nested: FileNotFoundException[/data/elasticsearch/nodes/0/indices/logstash-2012.05.17/3/index/\_6ui.nrm (Too many open files)]; ]]  
> [2012-05-18 03:43:55,098][INFO][node] [Evilhawk\<200e\>] {0.19.3}[14925]: stopping ...  
> [2012-05-18 03:43:55,104][INFO][river.rabbitmq] [Evilhawk\<200e\>] [rabbitmq][logstash-0\_0\_0\_0] closing rabbitmq river  
> [2012-05-18 03:44:16,928][INFO][node] [Terraxia] {0.19.3}[19822]: initializing ...  
> [2012-05-18 03:44:16,938][INFO][plugins] [Terraxia] loaded [river-rabbitmq], sites   
> [2012-05-18 03:44:20,420][INFO][node] [Terraxia] {0.19.3}[19822]: initialized  
> [2012-05-18 03:44:20,420][INFO][node] [Terraxia] {0.19.3}[19822]: starting ...  
> [2012-05-18 03:44:21,855][INFO][transport] [Terraxia] bound\_address {inet[/0.0.0.0:9301]}, publish\_address {inet[/10.170.110.119:9301]}  
> [2012-05-18 03:44:25,014][INFO][cluster.service] [Terraxia] detected\_master [Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]], added {[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]],[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason: zen-disco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
> [2012-05-18 03:44:29,454][INFO][discovery] [Terraxia] elasticsearch/2-Co3TxbRlqXHIhd597D\_A  
> [2012-05-18 03:44:29,460][INFO][http] [Terraxia] bound\_address {inet[/0.0.0.0:9201]}, publish\_address {inet[/10.170.110.119:9201]}  
> [2012-05-18 03:44:29,460][INFO][node] [Terraxia] {0.19.3}[19822]: started  
> [2012-05-18 03:44:35,864][INFO][node] [Evilhawk\<200e\>] {0.19.3}[14925]: stopped  
> [2012-05-18 03:44:35,864][INFO][node] [Evilhawk\<200e\>] {0.19.3}[14925]: closing ...  
> [2012-05-18 03:44:45,878][INFO][node] [Evilhawk\<200e\>] {0.19.3}[14925]: closed  
> [2012-05-18 03:45:00,834][INFO][cluster.service] [Terraxia] added {[Psi-Lord][kJv\_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],}, reason: zen-disco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
> [2012-05-18 03:47:35,093][INFO][cluster.service] [Terraxia] removed {[Psi-Lord][kJv\_bK5ESjSF7ihHurpnww][inet[/10.170.110.130:9301]],}, reason: zen-disco-receive(from master [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]])  
> [2012-05-18 22:14:18,472][INFO][discovery.zen] [Terraxia] master\_left [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reason [shut\_down]  
> [2012-05-18 22:14:18,500][INFO][discovery.zen] [Terraxia] master\_left [[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]], reason [transport disconnected (with verified connect)]  
> [2012-05-18 22:14:18,500][INFO][cluster.service] [Terraxia] master {new [Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]], previous [Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]]}, removed {[Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]],}, reason: zen-disco-master\_failed ([Newell, Walter][iahaWZ2fRpC05v8JHjtn-w][inet[/10.170.110.118:9300]])  
> [2012-05-18 22:14:54,233][INFO][cluster.service] [Terraxia] added {[Loss][FXpkOT72Rfii02DRD6BPSQ][inet[/10.170.110.118:9300]],}, reason: zen-disco-receive(from master [[Cheetah][0KbyW4qQRHi-dGttyMwSww][inet[/10.170.110.119:9300]]])

---

<div class="post-metadata">

**Author:** ![pedro\_goncalves](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pedro_goncalves/32/80461_2.png) [@pedro\_goncalves](https://discuss.elastic.co/u/pedro_goncalves)\
**Post date:** [September 4, 2013, 8:56pm UTC](https://discuss.elastic.co/t/river-flows-for-a-while-then-stops-with-exception/7763/4 "2013-09-04T20:56:06Z")

</div>

Hi dadoonet, that link seems to be dead (got a 404).

Can you please point to a reference or explain what do you mean with «too many open files is a "common" error» ?

Thanks!  
Pedro

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:18am UTC](https://discuss.elastic.co/t/river-flows-for-a-while-then-stops-with-exception/7763/5 "2017-07-06T02:18:10Z")

</div>