# Strange no. of file descriptors eating up

**URL:** <https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468>\
**Category:** Elasticsearch\
**Created:** [September 5, 2013, 4:01am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468 "2013-09-05T04:01:36Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kailashnath\_Reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailashnath_reddy/32/2124_2.png) [@Kailashnath\_Reddy](https://discuss.elastic.co/u/Kailashnath_Reddy)\
**Post date:** [September 5, 2013, 4:01am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/1 "2013-09-05T04:01:36Z")

</div>

Recently we've been running migration and we see a lot of errors with  
elasticsearch, after sometime it just doesn't accept any more connections.

All I see in the logs is:

Caused by: java.io.FileNotFoundException:  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx  
(Too many open files)  
at java.io.RandomAccessFile.open(Native Method)  
at java.io.RandomAccessFile.(RandomAccessFile.java:233)  
at  
org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:409)  
at  
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:123)  
at  
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)  
at  
org.apache.lucene.store.RateLimitedFSDirectory.openInput(RateLimitedFSDirectory.java:99)  
at  
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:436)  
at  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:98)  
at  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)  
at  
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:147)  
at  
org.apache.lucene.index.SegmentReader.(SegmentReader.java:56)  
at  
org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)  
at  
org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218)  
at  
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:96)  
at  
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:369)  
at  
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)  
at  
org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)  
at  
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1412)  
at  
org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:271)  
... 7 more  
[2013-09-05 05:52:11,722][WARN][cluster.action.shard] [Slave] sending  
failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R],  
s[INITIALIZING], reason [Failed to start shard, message  
[RecoveryFailedException[[verbose][4]: Recovery failed from  
[Master][4Zno5nzmQFqdh\_o5TbGf5A][inet[/10.223.12.31:9300]]{master=true}  
into  
[Slave][mFU1CeLwSr6sa4Jb0sqZpQ][inet[/10.223.12.32:9300]]{master=false}];  
nested:  
RemoteTransportException[[Master][inet[/10.223.12.31:9300]][index/shard/recovery/startRecovery]];  
nested: RecoveryEngineException[[verbose][4] Phase[2] Execution failed];  
nested:  
RemoteTransportException[[Slave][inet[/10.223.12.32:9300]][index/shard/recovery/prepareTranslog]];  
nested: EngineCreationFailureException[[verbose][4] failed to open reader  
on writer]; nested:  
FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx  
(Too many open files)]; ]]  
FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx  
(Too many open files)]; ]]

The ulimit in production machines are 65535 and when I try to do lsof -a -p  
, I see something like this:

6291457  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/translog/translog-1369668960033  
java 21548 kailashnath 848r REG 8,17 1941 10010832  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.pos  
java 21548 kailashnath 849r REG 8,17 15174 9971253  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.nvd  
java 21548 kailashnath 850r REG 8,17 114071 9971246  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.doc  
java 21548 kailashnath 851r REG 8,17 2090 10010846  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.pay  
java 21548 kailashnath 852r REG 8,17 606 10010847  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.tim  
java 21548 kailashnath 853r REG 8,17 855 10010690  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h.fdt  
java 21548 kailashnath 854r REG 8,17 899 9970974  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mda.fdx  
java 21548 kailashnath 855r REG 8,17 47 10010689  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h.fdx  
java 21548 kailashnath 856r REG 8,17 162098 10010696  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.pos  
java 21548 kailashnath 857r REG 8,17 34 10010702  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.pay  
java 21548 kailashnath 858r REG 8,17 391138 10010703  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.tim  
java 21548 kailashnath 859r REG 8,17 389565 10010694  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p.fdt  
java 21548 kailashnath 860r REG 8,17 158 10010693  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p.fdx  
java 21548 kailashnath 861r REG 8,17 400 10010915  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.nvd  
java 21548 kailashnath 862r REG 8,17 1183 10010905  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.doc  
java 21548 kailashnath 863r REG 8,17 2578 10010906  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.pos  
java 21548 kailashnath 864r REG 8,17 34 10010907  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.pay  
java 21548 kailashnath 865r REG 8,17 517 10010912  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.tim  
java 21548 kailashnath 866r REG 8,17 590 10010692  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.fdt  
java 21548 kailashnath 867r REG 8,17 45 10010691  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.fdx  
java 21548 kailashnath 869r REG 8,17 412 9971197  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.nvd  
java 21548 kailashnath 870r REG 8,17 4116 9971130  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.doc  
java 21548 kailashnath 871r REG 8,17 4306 9971133  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.pos  
java 21548 kailashnath 872r REG 8,17 34 9971191  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.pay  
java 21548 kailashnath 873r REG 8,17 13628 9971192  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.tim  
java 21548 kailashnath 874r REG 8,17 19938 9971127  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.fdt  
java 21548 kailashnath 875r REG 8,17 55 9971117  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.fdx  
java 21548 kailashnath 876r REG 8,17 136468 9971247  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.pos  
java 21548 kailashnath 877r REG 8,17 34 9971248  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.pay  
java 21548 kailashnath 878r REG 8,17 235767 9971249  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.tim  
java 21548 kailashnath 879r REG 8,17 670494 9971245  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.fdt  
java 21548 kailashnath 880r REG 8,17 736 9971244  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.fdx  
java 21548 kailashnath 881r REG 8,17 424 9971270  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.nvd  
java 21548 kailashnath 882r REG 8,17 4285 9971242  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.doc  
java 21548 kailashnath 883r REG 8,17 4441 9971243  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.pos  
java 21548 kailashnath 884r REG 8,17 34 9971264  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.pay  
java 21548 kailashnath 885r REG 8,17 14708 9971266  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.tim  
java 21548 kailashnath 886r REG 8,17 21322 9971060  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.fdt  
java 21548 kailashnath 887r REG 8,17 54 9971059  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.fdx  
java 21548 kailashnath 889r REG 8,17 16262 9970705  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.nvd  
java 21548 kailashnath 890r REG 8,17 155969 9970687  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.doc  
java 21548 kailashnath 910r REG 8,17 159939 9970689  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.pos  
java 21548 kailashnath 911r REG 8,17 34 9970690  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.pay  
java 21548 kailashnath 912r REG 8,17 341259 9970693  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.tim  
java 21548 kailashnath 913r REG 8,17 821871 9970644  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.fdt  
java 21548 kailashnath 914r REG 8,17 804 9970641  
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.fdx

Not sure what is the issue

/K

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 5, 2013, 4:57am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/2 "2013-09-05T04:57:33Z")

</div>

How many shards do you have on this node?

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 sept. 2013 à 06:01, Kailashnath Reddy [retadedgeek@gmail.com](mailto:retadedgeek@gmail.com) a écrit :

Recently we've been running migration and we see a lot of errors with elasticsearch, after sometime it just doesn't accept any more connections.

All I see in the logs is:

Caused by: java.io.FileNotFoundException: /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx (Too many open files)  
at java.io.RandomAccessFile.open(Native Method)  
at java.io.RandomAccessFile.(RandomAccessFile.java:233)  
at org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:409)  
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:123)  
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)  
at org.apache.lucene.store.RateLimitedFSDirectory.openInput(RateLimitedFSDirectory.java:99)  
at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:436)  
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:98)  
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)  
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:147)  
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:56)  
at org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)  
at org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218)  
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:96)  
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:369)  
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)  
at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)  
at org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1412)  
at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:271)  
... 7 more  
[2013-09-05 05:52:11,722][WARN][cluster.action.shard] [Slave] sending failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[[verbose][4]: Recovery failed from [Master][4Zno5nzmQFqdh\_o5TbGf5A][inet[/10.223.12.31:9300]]{master=true} into [Slave][mFU1CeLwSr6sa4Jb0sqZpQ][inet[/10.223.12.32:9300]]{master=false}]; nested: RemoteTransportException[[Master][inet[/10.223.12.31:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[verbose][4] Phase[2] Execution failed]; nested: RemoteTransportException[[Slave][inet[/10.223.12.32:9300]][index/shard/recovery/prepareTranslog]]; nested: EngineCreationFailureException[[verbose][4] failed to open reader on writer]; nested: FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx (Too many open files)]; ]]  
FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx (Too many open files)]; ]]

The ulimit in production machines are 65535 and when I try to do lsof -a -p , I see something like this:

6291457 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/translog/translog-1369668960033  
java 21548 kailashnath 848r REG 8,17 1941 10010832 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.pos  
java 21548 kailashnath 849r REG 8,17 15174 9971253 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.nvd  
java 21548 kailashnath 850r REG 8,17 114071 9971246 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.doc  
java 21548 kailashnath 851r REG 8,17 2090 10010846 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.pay  
java 21548 kailashnath 852r REG 8,17 606 10010847 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.tim  
java 21548 kailashnath 853r REG 8,17 855 10010690 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h.fdt  
java 21548 kailashnath 854r REG 8,17 899 9970974 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mda.fdx  
java 21548 kailashnath 855r REG 8,17 47 10010689 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h.fdx  
java 21548 kailashnath 856r REG 8,17 162098 10010696 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.pos  
java 21548 kailashnath 857r REG 8,17 34 10010702 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.pay  
java 21548 kailashnath 858r REG 8,17 391138 10010703 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.tim  
java 21548 kailashnath 859r REG 8,17 389565 10010694 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p.fdt  
java 21548 kailashnath 860r REG 8,17 158 10010693 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p.fdx  
java 21548 kailashnath 861r REG 8,17 400 10010915 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.nvd  
java 21548 kailashnath 862r REG 8,17 1183 10010905 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.doc  
java 21548 kailashnath 863r REG 8,17 2578 10010906 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.pos  
java 21548 kailashnath 864r REG 8,17 34 10010907 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.pay  
java 21548 kailashnath 865r REG 8,17 517 10010912 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.tim  
java 21548 kailashnath 866r REG 8,17 590 10010692 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.fdt  
java 21548 kailashnath 867r REG 8,17 45 10010691 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.fdx  
java 21548 kailashnath 869r REG 8,17 412 9971197 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.nvd  
java 21548 kailashnath 870r REG 8,17 4116 9971130 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.doc  
java 21548 kailashnath 871r REG 8,17 4306 9971133 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.pos  
java 21548 kailashnath 872r REG 8,17 34 9971191 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.pay  
java 21548 kailashnath 873r REG 8,17 13628 9971192 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.tim  
java 21548 kailashnath 874r REG 8,17 19938 9971127 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.fdt  
java 21548 kailashnath 875r REG 8,17 55 9971117 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.fdx  
java 21548 kailashnath 876r REG 8,17 136468 9971247 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.pos  
java 21548 kailashnath 877r REG 8,17 34 9971248 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.pay  
java 21548 kailashnath 878r REG 8,17 235767 9971249 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.tim  
java 21548 kailashnath 879r REG 8,17 670494 9971245 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.fdt  
java 21548 kailashnath 880r REG 8,17 736 9971244 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.fdx  
java 21548 kailashnath 881r REG 8,17 424 9971270 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.nvd  
java 21548 kailashnath 882r REG 8,17 4285 9971242 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.doc  
java 21548 kailashnath 883r REG 8,17 4441 9971243 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.pos  
java 21548 kailashnath 884r REG 8,17 34 9971264 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.pay  
java 21548 kailashnath 885r REG 8,17 14708 9971266 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.tim  
java 21548 kailashnath 886r REG 8,17 21322 9971060 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.fdt  
java 21548 kailashnath 887r REG 8,17 54 9971059 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.fdx  
java 21548 kailashnath 889r REG 8,17 16262 9970705 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.nvd  
java 21548 kailashnath 890r REG 8,17 155969 9970687 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.doc  
java 21548 kailashnath 910r REG 8,17 159939 9970689 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.pos  
java 21548 kailashnath 911r REG 8,17 34 9970690 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.pay  
java 21548 kailashnath 912r REG 8,17 341259 9970693 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.tim  
java 21548 kailashnath 913r REG 8,17 821871 9970644 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.fdt  
java 21548 kailashnath 914r REG 8,17 804 9970641 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.fdx

Not sure what is the issue

/K

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Kailashnath\_Reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailashnath_reddy/32/2124_2.png) [@Kailashnath\_Reddy](https://discuss.elastic.co/u/Kailashnath_Reddy)\
**Post date:** [September 5, 2013, 6:46am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/3 "2013-09-05T06:46:27Z")

</div>

Hi David,  
It's currently set to default 5.

# Set the number of shards (splits) of an index (5 by default):

# 

# index.number\_of\_shards: 5

# Set the number of replicas (additional copies) of an index (1 by default):

# 

index.number\_of\_replicas: 1

# Note, that for development on a local machine, with small indices, it

usually

# makes sense to "disable" the distributed features:

# 

# index.number\_of\_shards: 1

# index.number\_of\_replicas: 0

/K

On Thursday, September 5, 2013 10:27:33 AM UTC+5:30, David Pilato wrote:

> How many shards do you have on this node?
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 5 sept. 2013 à 06:01, Kailashnath Reddy \<[retad...@gmail.com](mailto:retad...@gmail.com)\<javascript:\>\>  
> a écrit :
> 
> Recently we've been running migration and we see a lot of errors with  
> elasticsearch, after sometime it just doesn't accept any more connections.
> 
> All I see in the logs is:
> 
> Caused by: java.io.FileNotFoundException:  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx  
> (Too many open files)  
> at java.io.RandomAccessFile.open(Native Method)  
> at java.io.RandomAccessFile.(RandomAccessFile.java:233)  
> at  
> org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:409)  
> at  
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:123)  
> at  
> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)  
> at  
> org.apache.lucene.store.RateLimitedFSDirectory.openInput(RateLimitedFSDirectory.java:99)  
> at  
> org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:436)  
> at  
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:98)  
> at  
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)  
> at  
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:147)  
> at  
> org.apache.lucene.index.SegmentReader.(SegmentReader.java:56)  
> at  
> org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)  
> at  
> org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218)  
> at  
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:96)  
> at  
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:369)  
> at  
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)  
> at  
> org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)  
> at  
> org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1412)  
> at  
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:271)  
> ... 7 more  
> [2013-09-05 05:52:11,722][WARN][cluster.action.shard] [Slave]  
> sending failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R],  
> s[INITIALIZING], reason [Failed to start shard, message  
> [RecoveryFailedException[[verbose][4]: Recovery failed from  
> [Master][4Zno5nzmQFqdh\_o5TbGf5A][inet[/10.223.12.31:9300]]{master=true}  
> into  
> [Slave][mFU1CeLwSr6sa4Jb0sqZpQ][inet[/10.223.12.32:9300]]{master=false}];  
> nested:  
> RemoteTransportException[[Master][inet[/10.223.12.31:9300]][index/shard/recovery/startRecovery]];  
> nested: RecoveryEngineException[[verbose][4] Phase[2] Execution failed];  
> nested:  
> RemoteTransportException[[Slave][inet[/10.223.12.32:9300]][index/shard/recovery/prepareTranslog]];  
> nested: EngineCreationFailureException[[verbose][4] failed to open reader  
> on writer]; nested:  
> FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx  
> (Too many open files)]; ]]  
> FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx  
> (Too many open files)]; ]]
> 
> The ulimit in production machines are 65535 and when I try to do lsof -a  
> -p , I see something like this:
> 
> 6291457  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/translog/translog-1369668960033  
> java 21548 kailashnath 848r REG 8,17 1941  
> 10010832  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.pos  
> java 21548 kailashnath 849r REG 8,17 15174  
> 9971253  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.nvd  
> java 21548 kailashnath 850r REG 8,17 114071  
> 9971246  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.doc  
> java 21548 kailashnath 851r REG 8,17 2090  
> 10010846  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.pay  
> java 21548 kailashnath 852r REG 8,17 606  
> 10010847  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.tim  
> java 21548 kailashnath 853r REG 8,17 855  
> 10010690  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h.fdt  
> java 21548 kailashnath 854r REG 8,17 899  
> 9970974  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mda.fdx  
> java 21548 kailashnath 855r REG 8,17 47  
> 10010689  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h.fdx  
> java 21548 kailashnath 856r REG 8,17 162098  
> 10010696  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.pos  
> java 21548 kailashnath 857r REG 8,17 34  
> 10010702  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.pay  
> java 21548 kailashnath 858r REG 8,17 391138  
> 10010703  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.tim  
> java 21548 kailashnath 859r REG 8,17 389565  
> 10010694  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p.fdt  
> java 21548 kailashnath 860r REG 8,17 158  
> 10010693  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p.fdx  
> java 21548 kailashnath 861r REG 8,17 400  
> 10010915  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.nvd  
> java 21548 kailashnath 862r REG 8,17 1183  
> 10010905  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.doc  
> java 21548 kailashnath 863r REG 8,17 2578  
> 10010906  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.pos  
> java 21548 kailashnath 864r REG 8,17 34  
> 10010907  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.pay  
> java 21548 kailashnath 865r REG 8,17 517  
> 10010912  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.tim  
> java 21548 kailashnath 866r REG 8,17 590  
> 10010692  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.fdt  
> java 21548 kailashnath 867r REG 8,17 45  
> 10010691  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.fdx  
> java 21548 kailashnath 869r REG 8,17 412  
> 9971197  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.nvd  
> java 21548 kailashnath 870r REG 8,17 4116  
> 9971130  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.doc  
> java 21548 kailashnath 871r REG 8,17 4306  
> 9971133  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.pos  
> java 21548 kailashnath 872r REG 8,17 34  
> 9971191  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.pay  
> java 21548 kailashnath 873r REG 8,17 13628  
> 9971192  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.tim  
> java 21548 kailashnath 874r REG 8,17 19938  
> 9971127  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.fdt  
> java 21548 kailashnath 875r REG 8,17 55  
> 9971117  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.fdx  
> java 21548 kailashnath 876r REG 8,17 136468  
> 9971247  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.pos  
> java 21548 kailashnath 877r REG 8,17 34  
> 9971248  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.pay  
> java 21548 kailashnath 878r REG 8,17 235767  
> 9971249  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.tim  
> java 21548 kailashnath 879r REG 8,17 670494  
> 9971245  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.fdt  
> java 21548 kailashnath 880r REG 8,17 736  
> 9971244  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.fdx  
> java 21548 kailashnath 881r REG 8,17 424  
> 9971270  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.nvd  
> java 21548 kailashnath 882r REG 8,17 4285  
> 9971242  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.doc  
> java 21548 kailashnath 883r REG 8,17 4441  
> 9971243  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.pos  
> java 21548 kailashnath 884r REG 8,17 34  
> 9971264  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.pay  
> java 21548 kailashnath 885r REG 8,17 14708  
> 9971266  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.tim  
> java 21548 kailashnath 886r REG 8,17 21322  
> 9971060  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.fdt  
> java 21548 kailashnath 887r REG 8,17 54  
> 9971059  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.fdx  
> java 21548 kailashnath 889r REG 8,17 16262  
> 9970705  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.nvd  
> java 21548 kailashnath 890r REG 8,17 155969  
> 9970687  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.doc  
> java 21548 kailashnath 910r REG 8,17 159939  
> 9970689  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.pos  
> java 21548 kailashnath 911r REG 8,17 34  
> 9970690  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.pay  
> java 21548 kailashnath 912r REG 8,17 341259  
> 9970693  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.tim  
> java 21548 kailashnath 913r REG 8,17 821871  
> 9970644  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.fdt  
> java 21548 kailashnath 914r REG 8,17 804  
> 9970641  
> /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.fdx
> 
> Not sure what is the issue
> 
> /K
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 5, 2013, 6:48am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/4 "2013-09-05T06:48:41Z")

</div>

How many index do you have?

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 sept. 2013 à 08:46, Kailashnath Reddy [retadedgeek@gmail.com](mailto:retadedgeek@gmail.com) a écrit :

> Hi David,  
> It's currently set to default 5.
> 
> # Set the number of shards (splits) of an index (5 by default):
> 
> # 
> 
> # index.number\_of\_shards: 5
> 
> # Set the number of replicas (additional copies) of an index (1 by default):
> 
> # 
> 
> index.number\_of\_replicas: 1
> 
> # Note, that for development on a local machine, with small indices, it usually
> 
> # makes sense to "disable" the distributed features:
> 
> # 
> 
> # index.number\_of\_shards: 1
> 
> # index.number\_of\_replicas: 0
> 
> /K
> 
> On Thursday, September 5, 2013 10:27:33 AM UTC+5:30, David Pilato wrote:
> 
> > How many shards do you have on this node?
> > 
> > --  
> > David 😉  
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > 
> > Le 5 sept. 2013 à 06:01, Kailashnath Reddy [retad...@gmail.com](mailto:retad...@gmail.com) a écrit :
> > 
> > Recently we've been running migration and we see a lot of errors with elasticsearch, after sometime it just doesn't accept any more connections.
> > 
> > All I see in the logs is:
> > 
> > Caused by: java.io.FileNotFoundException: /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx (Too many open files)  
> > at java.io.RandomAccessFile.open(Native Method)  
> > at java.io.RandomAccessFile.(RandomAccessFile.java:233)  
> > at org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:409)  
> > at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:123)  
> > at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)  
> > at org.apache.lucene.store.RateLimitedFSDirectory.openInput(RateLimitedFSDirectory.java:99)  
> > at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:436)  
> > at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:98)  
> > at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)  
> > at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:147)  
> > at org.apache.lucene.index.SegmentReader.(SegmentReader.java:56)  
> > at org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)  
> > at org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218)  
> > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:96)  
> > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:369)  
> > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)  
> > at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)  
> > at org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1412)  
> > at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:271)  
> > ... 7 more  
> > [2013-09-05 05:52:11,722][WARN][cluster.action.shard] [Slave] sending failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[[verbose][4]: Recovery failed from [Master][4Zno5nzmQFqdh\_o5TbGf5A][inet[/10.223.12.31:9300]]{master=true} into [Slave][mFU1CeLwSr6sa4Jb0sqZpQ][inet[/10.223.12.32:9300]]{master=false}]; nested: RemoteTransportException[[Master][inet[/10.223.12.31:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[verbose][4] Phase[2] Execution failed]; nested: RemoteTransportException[[Slave][inet[/10.223.12.32:9300]][index/shard/recovery/prepareTranslog]]; nested: EngineCreationFailureException[[verbose][4] failed to open reader on writer]; nested: FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx (Too many open files)]; ]]  
> > FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/\_6t86.fdx (Too many open files)]; ]]
> > 
> > The ulimit in production machines are 65535 and when I try to do lsof -a -p , I see something like this:
> > 
> > 6291457 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/translog/translog-1369668960033  
> > java 21548 kailashnath 848r REG 8,17 1941 10010832 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.pos  
> > java 21548 kailashnath 849r REG 8,17 15174 9971253 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.nvd  
> > java 21548 kailashnath 850r REG 8,17 114071 9971246 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.doc  
> > java 21548 kailashnath 851r REG 8,17 2090 10010846 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.pay  
> > java 21548 kailashnath 852r REG 8,17 606 10010847 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h\_es090\_0.tim  
> > java 21548 kailashnath 853r REG 8,17 855 10010690 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h.fdt  
> > java 21548 kailashnath 854r REG 8,17 899 9970974 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mda.fdx  
> > java 21548 kailashnath 855r REG 8,17 47 10010689 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/\_536h.fdx  
> > java 21548 kailashnath 856r REG 8,17 162098 10010696 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.pos  
> > java 21548 kailashnath 857r REG 8,17 34 10010702 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.pay  
> > java 21548 kailashnath 858r REG 8,17 391138 10010703 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p\_es090\_0.tim  
> > java 21548 kailashnath 859r REG 8,17 389565 10010694 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p.fdt  
> > java 21548 kailashnath 860r REG 8,17 158 10010693 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578p.fdx  
> > java 21548 kailashnath 861r REG 8,17 400 10010915 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.nvd  
> > java 21548 kailashnath 862r REG 8,17 1183 10010905 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.doc  
> > java 21548 kailashnath 863r REG 8,17 2578 10010906 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.pos  
> > java 21548 kailashnath 864r REG 8,17 34 10010907 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.pay  
> > java 21548 kailashnath 865r REG 8,17 517 10010912 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q\_es090\_0.tim  
> > java 21548 kailashnath 866r REG 8,17 590 10010692 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.fdt  
> > java 21548 kailashnath 867r REG 8,17 45 10010691 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/\_578q.fdx  
> > java 21548 kailashnath 869r REG 8,17 412 9971197 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.nvd  
> > java 21548 kailashnath 870r REG 8,17 4116 9971130 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.doc  
> > java 21548 kailashnath 871r REG 8,17 4306 9971133 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.pos  
> > java 21548 kailashnath 872r REG 8,17 34 9971191 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.pay  
> > java 21548 kailashnath 873r REG 8,17 13628 9971192 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36\_es090\_0.tim  
> > java 21548 kailashnath 874r REG 8,17 19938 9971127 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.fdt  
> > java 21548 kailashnath 875r REG 8,17 55 9971117 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/\_9w36.fdx  
> > java 21548 kailashnath 876r REG 8,17 136468 9971247 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.pos  
> > java 21548 kailashnath 877r REG 8,17 34 9971248 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.pay  
> > java 21548 kailashnath 878r REG 8,17 235767 9971249 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.tim  
> > java 21548 kailashnath 879r REG 8,17 670494 9971245 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.fdt  
> > java 21548 kailashnath 880r REG 8,17 736 9971244 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdk.fdx  
> > java 21548 kailashnath 881r REG 8,17 424 9971270 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.nvd  
> > java 21548 kailashnath 882r REG 8,17 4285 9971242 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.doc  
> > java 21548 kailashnath 883r REG 8,17 4441 9971243 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.pos  
> > java 21548 kailashnath 884r REG 8,17 34 9971264 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.pay  
> > java 21548 kailashnath 885r REG 8,17 14708 9971266 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl\_es090\_0.tim  
> > java 21548 kailashnath 886r REG 8,17 21322 9971060 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.fdt  
> > java 21548 kailashnath 887r REG 8,17 54 9971059 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/\_4mdl.fdx  
> > java 21548 kailashnath 889r REG 8,17 16262 9970705 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.nvd  
> > java 21548 kailashnath 890r REG 8,17 155969 9970687 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.doc  
> > java 21548 kailashnath 910r REG 8,17 159939 9970689 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.pos  
> > java 21548 kailashnath 911r REG 8,17 34 9970690 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.pay  
> > java 21548 kailashnath 912r REG 8,17 341259 9970693 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr\_es090\_0.tim  
> > java 21548 kailashnath 913r REG 8,17 821871 9970644 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.fdt  
> > java 21548 kailashnath 914r REG 8,17 804 9970641 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/\_9wdr.fdx
> > 
> > Not sure what is the issue
> > 
> > /K
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Kailashnath\_Reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailashnath_reddy/32/2124_2.png) [@Kailashnath\_Reddy](https://discuss.elastic.co/u/Kailashnath_Reddy)\
**Post date:** [September 5, 2013, 6:59am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/5 "2013-09-05T06:59:22Z")

</div>

Hi David,  
PFB the stats.  
There are two indices, "users" and "verbose" as shown in the below json!

{  
"ok" : true,  
"\_shards" : {  
"total" : 20,  
"successful" : 19,  
"failed" : 0  
},  
"\_all" : {  
"primaries" : {  
"docs" : {  
"count" : 7090542,  
"deleted" : 236979  
},  
"store" : {  
"size" : "1.2gb",  
"size\_in\_bytes" : 1376458012,  
"throttle\_time" : "0s",  
"throttle\_time\_in\_millis" : 0  
},  
"indexing" : {  
"index\_total" : 1328383,  
"index\_time" : "1.7m",  
"index\_time\_in\_millis" : 106482,  
"index\_current" : 0,  
"delete\_total" : 0,  
"delete\_time" : "0s",  
"delete\_time\_in\_millis" : 0,  
"delete\_current" : 0  
},  
"get" : {  
"total" : 0,  
"get\_time" : "0s",  
"time\_in\_millis" : 0,  
"exists\_total" : 0,  
"exists\_time" : "0s",  
"exists\_time\_in\_millis" : 0,  
"missing\_total" : 0,  
"missing\_time" : "0s",  
"missing\_time\_in\_millis" : 0,  
"current" : 0  
},  
"search" : {  
"open\_contexts" : 0,  
"query\_total" : 85,  
"query\_time" : "3.1s",  
"query\_time\_in\_millis" : 3135,  
"query\_current" : 0,  
"fetch\_total" : 32,  
"fetch\_time" : "92ms",  
"fetch\_time\_in\_millis" : 92,  
"fetch\_current" : 0  
}  
},  
"total" : {  
"docs" : {  
"count" : 12944611,  
"deleted" : 436068  
},  
"store" : {  
"size" : "2.3gb",  
"size\_in\_bytes" : 2506949989,  
"throttle\_time" : "0s",  
"throttle\_time\_in\_millis" : 0  
},  
"indexing" : {  
"index\_total" : 2512365,  
"index\_time" : "3.4m",  
"index\_time\_in\_millis" : 204570,  
"index\_current" : 0,  
"delete\_total" : 0,  
"delete\_time" : "0s",  
"delete\_time\_in\_millis" : 0,  
"delete\_current" : 0  
},  
"get" : {  
"total" : 0,  
"get\_time" : "0s",  
"time\_in\_millis" : 0,  
"exists\_total" : 0,  
"exists\_time" : "0s",  
"exists\_time\_in\_millis" : 0,  
"missing\_total" : 0,  
"missing\_time" : "0s",  
"missing\_time\_in\_millis" : 0,  
"current" : 0  
},  
"search" : {  
"open\_contexts" : 0,  
"query\_total" : 144,  
"query\_time" : "6.8s",  
"query\_time\_in\_millis" : 6867,  
"query\_current" : 0,  
"fetch\_total" : 53,  
"fetch\_time" : "360ms",  
"fetch\_time\_in\_millis" : 360,  
"fetch\_current" : 0  
}  
}  
},  
"indices" : {  
"users" : {  
"primaries" : {  
"docs" : {  
"count" : 60347,  
"deleted" : 5751  
},  
"store" : {  
"size" : "7.6mb",  
"size\_in\_bytes" : 7979376,  
"throttle\_time" : "0s",  
"throttle\_time\_in\_millis" : 0  
},  
"indexing" : {  
"index\_total" : 665000,  
"index\_time" : "30.3s",  
"index\_time\_in\_millis" : 30392,  
"index\_current" : 0,  
"delete\_total" : 0,  
"delete\_time" : "0s",  
"delete\_time\_in\_millis" : 0,  
"delete\_current" : 0  
},  
"get" : {  
"total" : 0,  
"get\_time" : "0s",  
"time\_in\_millis" : 0,  
"exists\_total" : 0,  
"exists\_time" : "0s",  
"exists\_time\_in\_millis" : 0,  
"missing\_total" : 0,  
"missing\_time" : "0s",  
"missing\_time\_in\_millis" : 0,  
"current" : 0  
},  
"search" : {  
"open\_contexts" : 0,  
"query\_total" : 0,  
"query\_time" : "0s",  
"query\_time\_in\_millis" : 0,  
"query\_current" : 0,  
"fetch\_total" : 0,  
"fetch\_time" : "0s",  
"fetch\_time\_in\_millis" : 0,  
"fetch\_current" : 0  
}  
},  
"total" : {  
"docs" : {  
"count" : 120691,  
"deleted" : 10869  
},  
"store" : {  
"size" : "16.2mb",  
"size\_in\_bytes" : 17090438,  
"throttle\_time" : "0s",  
"throttle\_time\_in\_millis" : 0  
},  
"indexing" : {  
"index\_total" : 1321446,  
"index\_time" : "58.7s",  
"index\_time\_in\_millis" : 58751,  
"index\_current" : 0,  
"delete\_total" : 0,  
"delete\_time" : "0s",  
"delete\_time\_in\_millis" : 0,  
"delete\_current" : 0  
},  
"get" : {  
"total" : 0,  
"get\_time" : "0s",  
"time\_in\_millis" : 0,  
"exists\_total" : 0,  
"exists\_time" : "0s",  
"exists\_time\_in\_millis" : 0,  
"missing\_total" : 0,  
"missing\_time" : "0s",  
"missing\_time\_in\_millis" : 0,  
"current" : 0  
},  
"search" : {  
"open\_contexts" : 0,  
"query\_total" : 0,  
"query\_time" : "0s",  
"query\_time\_in\_millis" : 0,  
"query\_current" : 0,  
"fetch\_total" : 0,  
"fetch\_time" : "0s",  
"fetch\_time\_in\_millis" : 0,  
"fetch\_current" : 0  
}  
}  
},  
"verbose" : {  
"primaries" : {  
"docs" : {  
"count" : 7030195,  
"deleted" : 231228  
},  
"store" : {  
"size" : "1.2gb",  
"size\_in\_bytes" : 1368478636,  
"throttle\_time" : "0s",  
"throttle\_time\_in\_millis" : 0  
},  
"indexing" : {  
"index\_total" : 663383,  
"index\_time" : "1.2m",  
"index\_time\_in\_millis" : 76090,  
"index\_current" : 0,  
"delete\_total" : 0,  
"delete\_time" : "0s",  
"delete\_time\_in\_millis" : 0,  
"delete\_current" : 0  
},  
"get" : {  
"total" : 0,  
"get\_time" : "0s",  
"time\_in\_millis" : 0,  
"exists\_total" : 0,  
"exists\_time" : "0s",  
"exists\_time\_in\_millis" : 0,  
"missing\_total" : 0,  
"missing\_time" : "0s",  
"missing\_time\_in\_millis" : 0,  
"current" : 0  
},  
"search" : {  
"open\_contexts" : 0,  
"query\_total" : 85,  
"query\_time" : "3.1s",  
"query\_time\_in\_millis" : 3135,  
"query\_current" : 0,  
"fetch\_total" : 32,  
"fetch\_time" : "92ms",  
"fetch\_time\_in\_millis" : 92,  
"fetch\_current" : 0  
}  
},  
"total" : {  
"docs" : {  
"count" : 12823920,  
"deleted" : 425199  
},  
"store" : {  
"size" : "2.3gb",  
"size\_in\_bytes" : 2489859551,  
"throttle\_time" : "0s",  
"throttle\_time\_in\_millis" : 0  
},  
"indexing" : {  
"index\_total" : 1190919,  
"index\_time" : "2.4m",  
"index\_time\_in\_millis" : 145819,  
"index\_current" : 0,  
"delete\_total" : 0,  
"delete\_time" : "0s",  
"delete\_time\_in\_millis" : 0,  
"delete\_current" : 0  
},  
"get" : {  
"total" : 0,  
"get\_time" : "0s",  
"time\_in\_millis" : 0,  
"exists\_total" : 0,  
"exists\_time" : "0s",  
"exists\_time\_in\_millis" : 0,  
"missing\_total" : 0,  
"missing\_time" : "0s",  
"missing\_time\_in\_millis" : 0,  
"current" : 0  
},  
"search" : {  
"open\_contexts" : 0,  
"query\_total" : 144,  
"query\_time" : "6.8s",  
"query\_time\_in\_millis" : 6867,  
"query\_current" : 0,  
"fetch\_total" : 53,  
"fetch\_time" : "360ms",  
"fetch\_time\_in\_millis" : 360,  
"fetch\_current" : 0  
}  
}  
}  
}  
}

/K

On Thu, Sep 5, 2013 at 12:18 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote:

> How many index do you have?
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 5 sept. 2013 à 08:46, Kailashnath Reddy [retadedgeek@gmail.com](mailto:retadedgeek@gmail.com) a  
> écrit :
> 
> Hi David,  
> It's currently set to default 5.
> 
> # Set the number of shards (splits) of an index (5 by default):
> 
> # 
> 
> # index.number\_of\_shards: 5
> 
> # Set the number of replicas (additional copies) of an index (1 by
> 
> default):
> 
> # 
> 
> index.number\_of\_replicas: 1
> 
> # Note, that for development on a local machine, with small indices, it
> 
> usually
> 
> # makes sense to "disable" the distributed features:
> 
> # 
> 
> # index.number\_of\_shards: 1
> 
> # index.number\_of\_replicas: 0
> 
> /K
> 
> On Thursday, September 5, 2013 10:27:33 AM UTC+5:30, David Pilato wrote:
> 
> > How many shards do you have on this node?
> > 
> > --  
> > David 😉  
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > 
> > Le 5 sept. 2013 à 06:01, Kailashnath Reddy [retad...@gmail.com](mailto:retad...@gmail.com) a écrit :
> > 
> > Recently we've been running migration and we see a lot of errors with  
> > elasticsearch, after sometime it just doesn't accept any more connections.
> > 
> > All I see in the logs is:
> > 
> > Caused by: java.io.FileNotFoundException: /opt/isystem/dbs/\*\*  
> > elasticsearch/Projectplace/\*\*nodes/0/indices/verbose/4/\*\*index/_6t86.fdx  
> > (Too many open files)  
> > at java.io.RandomAccessFile.open(\*\*Native Method)  
> > at java.io.RandomAccessFile.\<\*\*init\>(RandomAccessFile.java:\*\*233)  
> > at org.apache.lucene.store. **FSDirectory$FSIndexInput.\<**  
> > init\>(FSDirectory.java:409)  
> > at org.apache.lucene.store. **NIOFSDirectory$**  
> > NIOFSIndexInput.(\*\*NIOFSDirectory.java:123)  
> > at org.apache.lucene.store.**NIOFSDirectory.openInput(**  
> > NIOFSDirectory.java:80)  
> > at org.apache.lucene.store.\*\*RateLimitedFSDirectory.\*openInput(  
> > \*RateLimitedFSDirectory.java:**99)  
> > at org.elasticsearch.index.store.Store$StoreDirectory.  
> > openInput(Store.java:436)  
> > at org.apache.lucene.codecs.compressing.  
> > CompressingStoredFieldsReader.**(**CompressingStoredFieldsReader.**  
> > java:98)  
> > at org.apache.lucene.codecs. **compressing.**  
> > CompressingStoredFieldsFormat.**fieldsReader(**  
> > CompressingStoredFieldsFormat.\*\*java:113)  
> > at org.apache.lucene.index.**SegmentCoreReaders.(**  
> > SegmentCoreReaders.java:147)  
> > at org.apache.lucene.index.**SegmentReader.(**  
> > SegmentReader.java:56)  
> > at org.apache.lucene.index.**ReadersAndLiveDocs.getReader(**  
> > ReadersAndLiveDocs.java:121)  
> > at org.apache.lucene.index. **ReadersAndLiveDocs.**  
> > getReadOnlyClone(\*\*ReadersAndLiveDocs.java:218)  
> > at org.apache.lucene.index.**StandardDirectoryReader.open(**  
> > StandardDirectoryReader.java:\*\*96)  
> > at org.apache.lucene.index.**IndexWriter.getReader(**  
> > IndexWriter.java:369)  
> > at org.apache.lucene.index.**DirectoryReader.open(**  
> > DirectoryReader.java:111)  
> > at org.apache.lucene.search.**SearcherManager.(**  
> > SearcherManager.java:89)  
> > at org.elasticsearch.index. **engine.robin.RobinEngine.**  
> > buildSearchManager(\*\*RobinEngine.java:1412)  
> > at org.elasticsearch.index. **engine.robin.RobinEngine.**  
> > start(RobinEngine.java:271)  
> > ... 7 more  
> > [2013-09-05 05:52:11,722][WARN][cluster.action.shard] [Slave]  
> > sending failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R],  
> > s[INITIALIZING], reason [Failed to start shard, message  
> > [RecoveryFailedException[[\*\*verbose][4]: Recovery failed from  
> > [Master][4Zno5nzmQFqdh_\*\*o5TbGf5A][inet[/10.223.12.31:\*\*9300]]{master=true}  
> > into [Slave][\*\*mFU1CeLwSr6sa4Jb0sqZpQ][inet[/\*\*10.223.12.32:9300  
> > ]]{master=\*\*false}]; nested: RemoteTransportException[[\*\*Master][inet[/  
> > 10.223.12.31:\*\*9300]][index/shard/recovery/\*\*startRecovery]]; nested:  
> > RecoveryEngineException[[\*\*verbose][4] Phase[2] Execution failed];  
> > nested: RemoteTransportException[[**Slave][inet[/10.223.12.32:9300**  
> > ]][index/shard/recovery/**prepareTranslog]]; nested:  
> > EngineCreationFailureException**[[verbose][4] failed to open reader on  
> > writer]; nested: FileNotFoundException[/opt/\*_isystem/dbs/elasticsearch/_  
> > \*Projectplace/nodes/0/indices/\*\*verbose/4/index/\_6t86.fdx (Too many open  
> > files)]; ]]  
> > FileNotFoundException[/opt/ **isystem/dbs/elasticsearch/**  
> > Projectplace/nodes/0/indices/\*\*verbose/4/index/\_6t86.fdx (Too many open  
> > files)]; ]]
> > 
> > The ulimit in production machines are 65535 and when I try to do lsof -a  
> > -p , I see something like this:
> > 
> > 6291457 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/users/3/**translog/translog-1369668960033  
> > java 21548 kailashnath 848r REG 8,17 1941  
> > 10010832 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/3/index/\_536h\_es090\_0.pos  
> > java 21548 kailashnath 849r REG 8,17 15174  
> > 9971253 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/verbose/2/index/\_4mdk.nvd  
> > java 21548 kailashnath 850r REG 8,17 114071  
> > 9971246 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/verbose/2/index/\_4mdk\_es090\_0.doc  
> > java 21548 kailashnath 851r REG 8,17 2090  
> > 10010846 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/3/index/\_536h\_es090\_0.pay  
> > java 21548 kailashnath 852r REG 8,17 606  
> > 10010847 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/3/index/\_536h\_es090\_0.tim  
> > java 21548 kailashnath 853r REG 8,17 855  
> > 10010690 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/3/index/**\_536h.fdt  
> > java 21548 kailashnath 854r REG 8,17 899  
> > 9970974 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/**index/\_4mda.fdx  
> > java 21548 kailashnath 855r REG 8,17 47  
> > 10010689 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/3/index/**\_536h.fdx  
> > java 21548 kailashnath 856r REG 8,17 162098  
> > 10010696 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/users/1/index/**\_578p\_es090\_0.pos  
> > java 21548 kailashnath 857r REG 8,17 34  
> > 10010702 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/1/index/**\_578p\_es090\_0.pay  
> > java 21548 kailashnath 858r REG 8,17 391138  
> > 10010703 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/users/1/index/**\_578p\_es090\_0.tim  
> > java 21548 kailashnath 859r REG 8,17 389565  
> > 10010694 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/1/index/**\_578p.fdt  
> > java 21548 kailashnath 860r REG 8,17 158  
> > 10010693 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/users/1/index/**\_578p.fdx  
> > java 21548 kailashnath 861r REG 8,17 400  
> > 10010915 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/1/index/**\_578q.nvd  
> > java 21548 kailashnath 862r REG 8,17 1183  
> > 10010905 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/users/1/index/**\_578q\_es090\_0.doc  
> > java 21548 kailashnath 863r REG 8,17 2578  
> > 10010906 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/1/index/**\_578q\_es090\_0.pos  
> > java 21548 kailashnath 864r REG 8,17 34  
> > 10010907 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/users/1/index/**\_578q\_es090\_0.pay  
> > java 21548 kailashnath 865r REG 8,17 517  
> > 10010912 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/1/index/**\_578q\_es090\_0.tim  
> > java 21548 kailashnath 866r REG 8,17 590  
> > 10010692 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/users/1/index/**\_578q.fdt  
> > java 21548 kailashnath 867r REG 8,17 45  
> > 10010691 /opt/isystem/dbs/elasticsearch/Projectplace/  
> > nodes/0/indices/users/1/index/**\_578q.fdx  
> > java 21548 kailashnath 869r REG 8,17 412  
> > 9971197 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/1/\*\*index/\_9w36.nvd  
> > java 21548 kailashnath 870r REG 8,17 4116  
> > 9971130 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/1/\*\*index/\_9w36\_es090\_0.doc  
> > java 21548 kailashnath 871r REG 8,17 4306  
> > 9971133 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/1/\*\*index/\_9w36\_es090\_0.pos  
> > java 21548 kailashnath 872r REG 8,17 34  
> > 9971191 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/1/\*\*index/\_9w36\_es090\_0.pay  
> > java 21548 kailashnath 873r REG 8,17 13628  
> > 9971192 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/1/\*\*index/\_9w36\_es090\_0.tim  
> > java 21548 kailashnath 874r REG 8,17 19938  
> > 9971127 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/1/\*\*index/\_9w36.fdt  
> > java 21548 kailashnath 875r REG 8,17 55  
> > 9971117 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/1/\*\*index/\_9w36.fdx  
> > java 21548 kailashnath 876r REG 8,17 136468  
> > 9971247 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdk\_es090\_0.pos  
> > java 21548 kailashnath 877r REG 8,17 34  
> > 9971248 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdk\_es090\_0.pay  
> > java 21548 kailashnath 878r REG 8,17 235767  
> > 9971249 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdk\_es090\_0.tim  
> > java 21548 kailashnath 879r REG 8,17 670494  
> > 9971245 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdk.fdt  
> > java 21548 kailashnath 880r REG 8,17 736  
> > 9971244 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdk.fdx  
> > java 21548 kailashnath 881r REG 8,17 424  
> > 9971270 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdl.nvd  
> > java 21548 kailashnath 882r REG 8,17 4285  
> > 9971242 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdl\_es090\_0.doc  
> > java 21548 kailashnath 883r REG 8,17 4441  
> > 9971243 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdl\_es090\_0.pos  
> > java 21548 kailashnath 884r REG 8,17 34  
> > 9971264 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdl\_es090\_0.pay  
> > java 21548 kailashnath 885r REG 8,17 14708  
> > 9971266 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdl\_es090\_0.tim  
> > java 21548 kailashnath 886r REG 8,17 21322  
> > 9971060 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdl.fdt  
> > java 21548 kailashnath 887r REG 8,17 54  
> > 9971059 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/2/\*\*index/\_4mdl.fdx  
> > java 21548 kailashnath 889r REG 8,17 16262  
> > 9970705 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/3/\*\*index/\_9wdr.nvd  
> > java 21548 kailashnath 890r REG 8,17 155969  
> > 9970687 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/3/\*\*index/\_9wdr\_es090\_0.doc  
> > java 21548 kailashnath 910r REG 8,17 159939  
> > 9970689 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/3/\*\*index/\_9wdr\_es090\_0.pos  
> > java 21548 kailashnath 911r REG 8,17 34  
> > 9970690 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/3/\*\*index/\_9wdr\_es090\_0.pay  
> > java 21548 kailashnath 912r REG 8,17 341259  
> > 9970693 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/3/\*\*index/\_9wdr\_es090\_0.tim  
> > java 21548 kailashnath 913r REG 8,17 821871  
> > 9970644 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/3/\*\*index/\_9wdr.fdt  
> > java 21548 kailashnath 914r REG 8,17 804  
> > 9970641 /opt/isystem/dbs/ **elasticsearch/Projectplace/**  
> > nodes/0/indices/verbose/3/\*\*index/\_9wdr.fdx
> > 
> > Not sure what is the issue
> > 
> > /K
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).  
> > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > .
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> 
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit  
> [https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [September 5, 2013, 7:14am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/6 "2013-09-05T07:14:57Z")

</div>

Check with curl localhost:9200/\_cluster/nodes/stats?process the current  
number of open files.

Jörg

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Kailashnath\_Reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailashnath_reddy/32/2124_2.png) [@Kailashnath\_Reddy](https://discuss.elastic.co/u/Kailashnath_Reddy)\
**Post date:** [September 5, 2013, 7:44am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/7 "2013-09-05T07:44:53Z")

</div>

Hi Jorg,  
PFB the response. It seems to be stable now, but still I seriously  
doubt what could have been the reason.

Restarted both the master/slave ES instances!

{  
"cluster\_name": "xxxxxxxxxx",  
"nodes": {  
"rnlGU2e-QiOWACCtXsmi5Q": {  
"timestamp": 1378366972509,  
"name": "Slave",  
"transport\_address": "inet[/10.223.12.32:9300]",  
"hostname": "pp-feedsdb-2",  
"attributes": {  
"master": "false"  
},  
"indices": {  
"docs": {  
"count": 6020937,  
"deleted": 203431  
},  
"store": {  
"size": "1gb",  
"size\_in\_bytes": 1159436704,  
"throttle\_time": "0s",  
"throttle\_time\_in\_millis": 0  
},  
"indexing": {  
"index\_total": 1574046,  
"index\_time": "2.1m",  
"index\_time\_in\_millis": 130263,  
"index\_current": 0,  
"delete\_total": 0,  
"delete\_time": "0s",  
"delete\_time\_in\_millis": 0,  
"delete\_current": 0  
},  
"get": {  
"total": 0,  
"get\_time": "0s",  
"time\_in\_millis": 0,  
"exists\_total": 0,  
"exists\_time": "0s",  
"exists\_time\_in\_millis": 0,  
"missing\_total": 0,  
"missing\_time": "0s",  
"missing\_time\_in\_millis": 0,  
"current": 0  
},  
"search": {  
"open\_contexts": 0,  
"query\_total": 75,  
"query\_time": "4.4s",  
"query\_time\_in\_millis": 4412,  
"query\_current": 0,  
"fetch\_total": 31,  
"fetch\_time": "738ms",  
"fetch\_time\_in\_millis": 738,  
"fetch\_current": 0  
}  
},  
"process": {  
"timestamp": 1378366972509,  
"open\_file\_descriptors": 862,  
"cpu": {  
"percent": 1,  
"sys": "27 seconds and 950 milliseconds",  
"sys\_in\_millis": 27950,  
"user": "4 minutes, 51 seconds and 310 milliseconds",  
"user\_in\_millis": 291310,  
"total": "5 minutes, 19 seconds and 260 milliseconds",  
"total\_in\_millis": 319260  
},  
"mem": {  
"resident": "389.2mb",  
"resident\_in\_bytes": 408113152,  
"share": "10.1mb",  
"share\_in\_bytes": 10625024,  
"total\_virtual": "4.3gb",  
"total\_virtual\_in\_bytes": 4683698176  
}  
}  
},  
"FpLkq6O6SviFsA9PXINnWw": {  
"timestamp": 1378366972506,  
"name": "Master",  
"transport\_address": "inet[/10.223.12.31:9300]",  
"hostname": "pp-feedsdb-1",  
"attributes": {  
"master": "true"  
},  
"indices": {  
"docs": {  
"count": 7299347,  
"deleted": 241932  
},  
"store": {  
"size": "1.3gb",  
"size\_in\_bytes": 1408533151,  
"throttle\_time": "0s",  
"throttle\_time\_in\_millis": 0  
},  
"indexing": {  
"index\_total": 1745124,  
"index\_time": "2.2m",  
"index\_time\_in\_millis": 136104,  
"index\_current": 0,  
"delete\_total": 0,  
"delete\_time": "0s",  
"delete\_time\_in\_millis": 0,  
"delete\_current": 0  
},  
"get": {  
"total": 0,  
"get\_time": "0s",  
"time\_in\_millis": 0,  
"exists\_total": 0,  
"exists\_time": "0s",  
"exists\_time\_in\_millis": 0,  
"missing\_total": 0,  
"missing\_time": "0s",  
"missing\_time\_in\_millis": 0,  
"current": 0  
},  
"search": {  
"open\_contexts": 0,  
"query\_total": 110,  
"query\_time": "3.9s",  
"query\_time\_in\_millis": 3922,  
"query\_current": 0,  
"fetch\_total": 48,  
"fetch\_time": "159ms",  
"fetch\_time\_in\_millis": 159,  
"fetch\_current": 0  
}  
},  
"process": {  
"timestamp": 1378366972506,  
"open\_file\_descriptors": 1004,  
"cpu": {  
"percent": 2,  
"sys": "34 seconds and 370 milliseconds",  
"sys\_in\_millis": 34370,  
"user": "5 minutes, 28 seconds and 710 milliseconds",  
"user\_in\_millis": 328710,  
"total": "6 minutes, 3 seconds and 80 milliseconds",  
"total\_in\_millis": 363080  
},  
"mem": {  
"resident": "397.2mb",  
"resident\_in\_bytes": 416587776,  
"share": "10.2mb",  
"share\_in\_bytes": 10788864,  
"total\_virtual": "4.3gb",  
"total\_virtual\_in\_bytes": 4688175104  
}  
}  
}  
}  
}

On Thu, Sep 5, 2013 at 12:44 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \<  
[joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote:

> Check with curl localhost:9200/\_cluster/nodes/stats?process the current  
> number of open files.
> 
> Jörg
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit  
> [https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 5, 2013, 7:50am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/8 "2013-09-05T07:50:12Z")

</div>

Did you run an \_optimize or something?  
Or did you launch a big "Delete" query?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 5 sept. 2013 à 09:44, Kailash ź Reddy [retadedgeek@gmail.com](mailto:retadedgeek@gmail.com) a écrit :

> Hi Jorg,  
> PFB the response. It seems to be stable now, but still I seriously doubt what could have been the reason.
> 
> Restarted both the master/slave ES instances!
> 
> {  
> "cluster\_name": "xxxxxxxxxx",  
> "nodes": {  
> "rnlGU2e-QiOWACCtXsmi5Q": {  
> "timestamp": 1378366972509,  
> "name": "Slave",  
> "transport\_address": "inet[/10.223.12.32:9300]",  
> "hostname": "pp-feedsdb-2",  
> "attributes": {  
> "master": "false"  
> },  
> "indices": {  
> "docs": {  
> "count": 6020937,  
> "deleted": 203431  
> },  
> "store": {  
> "size": "1gb",  
> "size\_in\_bytes": 1159436704,  
> "throttle\_time": "0s",  
> "throttle\_time\_in\_millis": 0  
> },  
> "indexing": {  
> "index\_total": 1574046,  
> "index\_time": "2.1m",  
> "index\_time\_in\_millis": 130263,  
> "index\_current": 0,  
> "delete\_total": 0,  
> "delete\_time": "0s",  
> "delete\_time\_in\_millis": 0,  
> "delete\_current": 0  
> },  
> "get": {  
> "total": 0,  
> "get\_time": "0s",  
> "time\_in\_millis": 0,  
> "exists\_total": 0,  
> "exists\_time": "0s",  
> "exists\_time\_in\_millis": 0,  
> "missing\_total": 0,  
> "missing\_time": "0s",  
> "missing\_time\_in\_millis": 0,  
> "current": 0  
> },  
> "search": {  
> "open\_contexts": 0,  
> "query\_total": 75,  
> "query\_time": "4.4s",  
> "query\_time\_in\_millis": 4412,  
> "query\_current": 0,  
> "fetch\_total": 31,  
> "fetch\_time": "738ms",  
> "fetch\_time\_in\_millis": 738,  
> "fetch\_current": 0  
> }  
> },  
> "process": {  
> "timestamp": 1378366972509,  
> "open\_file\_descriptors": 862,  
> "cpu": {  
> "percent": 1,  
> "sys": "27 seconds and 950 milliseconds",  
> "sys\_in\_millis": 27950,  
> "user": "4 minutes, 51 seconds and 310 milliseconds",  
> "user\_in\_millis": 291310,  
> "total": "5 minutes, 19 seconds and 260 milliseconds",  
> "total\_in\_millis": 319260  
> },  
> "mem": {  
> "resident": "389.2mb",  
> "resident\_in\_bytes": 408113152,  
> "share": "10.1mb",  
> "share\_in\_bytes": 10625024,  
> "total\_virtual": "4.3gb",  
> "total\_virtual\_in\_bytes": 4683698176  
> }  
> }  
> },  
> "FpLkq6O6SviFsA9PXINnWw": {  
> "timestamp": 1378366972506,  
> "name": "Master",  
> "transport\_address": "inet[/10.223.12.31:9300]",  
> "hostname": "pp-feedsdb-1",  
> "attributes": {  
> "master": "true"  
> },  
> "indices": {  
> "docs": {  
> "count": 7299347,  
> "deleted": 241932  
> },  
> "store": {  
> "size": "1.3gb",  
> "size\_in\_bytes": 1408533151,  
> "throttle\_time": "0s",  
> "throttle\_time\_in\_millis": 0  
> },  
> "indexing": {  
> "index\_total": 1745124,  
> "index\_time": "2.2m",  
> "index\_time\_in\_millis": 136104,  
> "index\_current": 0,  
> "delete\_total": 0,  
> "delete\_time": "0s",  
> "delete\_time\_in\_millis": 0,  
> "delete\_current": 0  
> },  
> "get": {  
> "total": 0,  
> "get\_time": "0s",  
> "time\_in\_millis": 0,  
> "exists\_total": 0,  
> "exists\_time": "0s",  
> "exists\_time\_in\_millis": 0,  
> "missing\_total": 0,  
> "missing\_time": "0s",  
> "missing\_time\_in\_millis": 0,  
> "current": 0  
> },  
> "search": {  
> "open\_contexts": 0,  
> "query\_total": 110,  
> "query\_time": "3.9s",  
> "query\_time\_in\_millis": 3922,  
> "query\_current": 0,  
> "fetch\_total": 48,  
> "fetch\_time": "159ms",  
> "fetch\_time\_in\_millis": 159,  
> "fetch\_current": 0  
> }  
> },  
> "process": {  
> "timestamp": 1378366972506,  
> "open\_file\_descriptors": 1004,  
> "cpu": {  
> "percent": 2,  
> "sys": "34 seconds and 370 milliseconds",  
> "sys\_in\_millis": 34370,  
> "user": "5 minutes, 28 seconds and 710 milliseconds",  
> "user\_in\_millis": 328710,  
> "total": "6 minutes, 3 seconds and 80 milliseconds",  
> "total\_in\_millis": 363080  
> },  
> "mem": {  
> "resident": "397.2mb",  
> "resident\_in\_bytes": 416587776,  
> "share": "10.2mb",  
> "share\_in\_bytes": 10788864,  
> "total\_virtual": "4.3gb",  
> "total\_virtual\_in\_bytes": 4688175104  
> }  
> }  
> }  
> }  
> }
> 
> On Thu, Sep 5, 2013 at 12:44 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:  
> Check with curl localhost:9200/\_cluster/nodes/stats?process the current number of open files.
> 
> Jörg
> 
> --  
> You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Kailashnath\_Reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailashnath_reddy/32/2124_2.png) [@Kailashnath\_Reddy](https://discuss.elastic.co/u/Kailashnath_Reddy)\
**Post date:** [September 5, 2013, 7:56am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/9 "2013-09-05T07:56:41Z")

</div>

No, we didn't run any delete query or optimize at all and there are no  
delete possibilities in the code too!

/K

On Thu, Sep 5, 2013 at 1:20 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote:

> Did you run an \_optimize or something?  
> Or did you launch a big "Delete" query?
> 
> --  
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_  
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)  
> | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
> 
> Le 5 sept. 2013 à 09:44, Kailash ź Reddy [retadedgeek@gmail.com](mailto:retadedgeek@gmail.com) a écrit :
> 
> Hi Jorg,  
> PFB the response. It seems to be stable now, but still I seriously  
> doubt what could have been the reason.
> 
> Restarted both the master/slave ES instances!
> 
> {  
> "cluster\_name": "xxxxxxxxxx",  
> "nodes": {  
> "rnlGU2e-QiOWACCtXsmi5Q": {  
> "timestamp": 1378366972509,  
> "name": "Slave",  
> "transport\_address": "inet[/10.223.12.32:9300]",  
> "hostname": "pp-feedsdb-2",  
> "attributes": {  
> "master": "false"  
> },  
> "indices": {  
> "docs": {  
> "count": 6020937,  
> "deleted": 203431  
> },  
> "store": {  
> "size": "1gb",  
> "size\_in\_bytes": 1159436704,  
> "throttle\_time": "0s",  
> "throttle\_time\_in\_millis": 0  
> },  
> "indexing": {  
> "index\_total": 1574046,  
> "index\_time": "2.1m",  
> "index\_time\_in\_millis": 130263,  
> "index\_current": 0,  
> "delete\_total": 0,  
> "delete\_time": "0s",  
> "delete\_time\_in\_millis": 0,  
> "delete\_current": 0  
> },  
> "get": {  
> "total": 0,  
> "get\_time": "0s",  
> "time\_in\_millis": 0,  
> "exists\_total": 0,  
> "exists\_time": "0s",  
> "exists\_time\_in\_millis": 0,  
> "missing\_total": 0,  
> "missing\_time": "0s",  
> "missing\_time\_in\_millis": 0,  
> "current": 0  
> },  
> "search": {  
> "open\_contexts": 0,  
> "query\_total": 75,  
> "query\_time": "4.4s",  
> "query\_time\_in\_millis": 4412,  
> "query\_current": 0,  
> "fetch\_total": 31,  
> "fetch\_time": "738ms",  
> "fetch\_time\_in\_millis": 738,  
> "fetch\_current": 0  
> }  
> },  
> "process": {  
> "timestamp": 1378366972509,  
> "open\_file\_descriptors": 862,  
> "cpu": {  
> "percent": 1,  
> "sys": "27 seconds and 950 milliseconds",  
> "sys\_in\_millis": 27950,  
> "user": "4 minutes, 51 seconds and 310 milliseconds",  
> "user\_in\_millis": 291310,  
> "total": "5 minutes, 19 seconds and 260 milliseconds",  
> "total\_in\_millis": 319260  
> },  
> "mem": {  
> "resident": "389.2mb",  
> "resident\_in\_bytes": 408113152,  
> "share": "10.1mb",  
> "share\_in\_bytes": 10625024,  
> "total\_virtual": "4.3gb",  
> "total\_virtual\_in\_bytes": 4683698176  
> }  
> }  
> },  
> "FpLkq6O6SviFsA9PXINnWw": {  
> "timestamp": 1378366972506,  
> "name": "Master",  
> "transport\_address": "inet[/10.223.12.31:9300]",  
> "hostname": "pp-feedsdb-1",  
> "attributes": {  
> "master": "true"  
> },  
> "indices": {  
> "docs": {  
> "count": 7299347,  
> "deleted": 241932  
> },  
> "store": {  
> "size": "1.3gb",  
> "size\_in\_bytes": 1408533151,  
> "throttle\_time": "0s",  
> "throttle\_time\_in\_millis": 0  
> },  
> "indexing": {  
> "index\_total": 1745124,  
> "index\_time": "2.2m",  
> "index\_time\_in\_millis": 136104,  
> "index\_current": 0,  
> "delete\_total": 0,  
> "delete\_time": "0s",  
> "delete\_time\_in\_millis": 0,  
> "delete\_current": 0  
> },  
> "get": {  
> "total": 0,  
> "get\_time": "0s",  
> "time\_in\_millis": 0,  
> "exists\_total": 0,  
> "exists\_time": "0s",  
> "exists\_time\_in\_millis": 0,  
> "missing\_total": 0,  
> "missing\_time": "0s",  
> "missing\_time\_in\_millis": 0,  
> "current": 0  
> },  
> "search": {  
> "open\_contexts": 0,  
> "query\_total": 110,  
> "query\_time": "3.9s",  
> "query\_time\_in\_millis": 3922,  
> "query\_current": 0,  
> "fetch\_total": 48,  
> "fetch\_time": "159ms",  
> "fetch\_time\_in\_millis": 159,  
> "fetch\_current": 0  
> }  
> },  
> "process": {  
> "timestamp": 1378366972506,  
> "open\_file\_descriptors": 1004,  
> "cpu": {  
> "percent": 2,  
> "sys": "34 seconds and 370 milliseconds",  
> "sys\_in\_millis": 34370,  
> "user": "5 minutes, 28 seconds and 710 milliseconds",  
> "user\_in\_millis": 328710,  
> "total": "6 minutes, 3 seconds and 80 milliseconds",  
> "total\_in\_millis": 363080  
> },  
> "mem": {  
> "resident": "397.2mb",  
> "resident\_in\_bytes": 416587776,  
> "share": "10.2mb",  
> "share\_in\_bytes": 10788864,  
> "total\_virtual": "4.3gb",  
> "total\_virtual\_in\_bytes": 4688175104  
> }  
> }  
> }  
> }  
> }
> 
> On Thu, Sep 5, 2013 at 12:44 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \<  
> [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote:
> 
> > Check with curl localhost:9200/\_cluster/nodes/stats?process the current  
> > number of open files.
> > 
> > Jörg
> > 
> > --  
> > You received this message because you are subscribed to a topic in the  
> > Google Groups "elasticsearch" group.  
> > To unsubscribe from this topic, visit  
> > [https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe).  
> > To unsubscribe from this group and all its topics, send an email to  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> 
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 5, 2013, 8:01am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/10 "2013-09-05T08:01:05Z")

</div>

But you have many updates then.  
It causes delete+insert behind the scene.

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 5 sept. 2013 à 09:56, Kailash ź Reddy [retadedgeek@gmail.com](mailto:retadedgeek@gmail.com) a écrit :

> No, we didn't run any delete query or optimize at all and there are no delete possibilities in the code too!
> 
> /K
> 
> On Thu, Sep 5, 2013 at 1:20 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote:  
> Did you run an \_optimize or something?  
> Or did you launch a big "Delete" query?
> 
> --  
> David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> @dadoonet | @elasticsearchfr | @scrutmydocs
> 
> Le 5 sept. 2013 à 09:44, Kailash ź Reddy [retadedgeek@gmail.com](mailto:retadedgeek@gmail.com) a écrit :
> 
> > Hi Jorg,  
> > PFB the response. It seems to be stable now, but still I seriously doubt what could have been the reason.
> > 
> > Restarted both the master/slave ES instances!
> > 
> > {  
> > "cluster\_name": "xxxxxxxxxx",  
> > "nodes": {  
> > "rnlGU2e-QiOWACCtXsmi5Q": {  
> > "timestamp": 1378366972509,  
> > "name": "Slave",  
> > "transport\_address": "inet[/10.223.12.32:9300]",  
> > "hostname": "pp-feedsdb-2",  
> > "attributes": {  
> > "master": "false"  
> > },  
> > "indices": {  
> > "docs": {  
> > "count": 6020937,  
> > "deleted": 203431  
> > },  
> > "store": {  
> > "size": "1gb",  
> > "size\_in\_bytes": 1159436704,  
> > "throttle\_time": "0s",  
> > "throttle\_time\_in\_millis": 0  
> > },  
> > "indexing": {  
> > "index\_total": 1574046,  
> > "index\_time": "2.1m",  
> > "index\_time\_in\_millis": 130263,  
> > "index\_current": 0,  
> > "delete\_total": 0,  
> > "delete\_time": "0s",  
> > "delete\_time\_in\_millis": 0,  
> > "delete\_current": 0  
> > },  
> > "get": {  
> > "total": 0,  
> > "get\_time": "0s",  
> > "time\_in\_millis": 0,  
> > "exists\_total": 0,  
> > "exists\_time": "0s",  
> > "exists\_time\_in\_millis": 0,  
> > "missing\_total": 0,  
> > "missing\_time": "0s",  
> > "missing\_time\_in\_millis": 0,  
> > "current": 0  
> > },  
> > "search": {  
> > "open\_contexts": 0,  
> > "query\_total": 75,  
> > "query\_time": "4.4s",  
> > "query\_time\_in\_millis": 4412,  
> > "query\_current": 0,  
> > "fetch\_total": 31,  
> > "fetch\_time": "738ms",  
> > "fetch\_time\_in\_millis": 738,  
> > "fetch\_current": 0  
> > }  
> > },  
> > "process": {  
> > "timestamp": 1378366972509,  
> > "open\_file\_descriptors": 862,  
> > "cpu": {  
> > "percent": 1,  
> > "sys": "27 seconds and 950 milliseconds",  
> > "sys\_in\_millis": 27950,  
> > "user": "4 minutes, 51 seconds and 310 milliseconds",  
> > "user\_in\_millis": 291310,  
> > "total": "5 minutes, 19 seconds and 260 milliseconds",  
> > "total\_in\_millis": 319260  
> > },  
> > "mem": {  
> > "resident": "389.2mb",  
> > "resident\_in\_bytes": 408113152,  
> > "share": "10.1mb",  
> > "share\_in\_bytes": 10625024,  
> > "total\_virtual": "4.3gb",  
> > "total\_virtual\_in\_bytes": 4683698176  
> > }  
> > }  
> > },  
> > "FpLkq6O6SviFsA9PXINnWw": {  
> > "timestamp": 1378366972506,  
> > "name": "Master",  
> > "transport\_address": "inet[/10.223.12.31:9300]",  
> > "hostname": "pp-feedsdb-1",  
> > "attributes": {  
> > "master": "true"  
> > },  
> > "indices": {  
> > "docs": {  
> > "count": 7299347,  
> > "deleted": 241932  
> > },  
> > "store": {  
> > "size": "1.3gb",  
> > "size\_in\_bytes": 1408533151,  
> > "throttle\_time": "0s",  
> > "throttle\_time\_in\_millis": 0  
> > },  
> > "indexing": {  
> > "index\_total": 1745124,  
> > "index\_time": "2.2m",  
> > "index\_time\_in\_millis": 136104,  
> > "index\_current": 0,  
> > "delete\_total": 0,  
> > "delete\_time": "0s",  
> > "delete\_time\_in\_millis": 0,  
> > "delete\_current": 0  
> > },  
> > "get": {  
> > "total": 0,  
> > "get\_time": "0s",  
> > "time\_in\_millis": 0,  
> > "exists\_total": 0,  
> > "exists\_time": "0s",  
> > "exists\_time\_in\_millis": 0,  
> > "missing\_total": 0,  
> > "missing\_time": "0s",  
> > "missing\_time\_in\_millis": 0,  
> > "current": 0  
> > },  
> > "search": {  
> > "open\_contexts": 0,  
> > "query\_total": 110,  
> > "query\_time": "3.9s",  
> > "query\_time\_in\_millis": 3922,  
> > "query\_current": 0,  
> > "fetch\_total": 48,  
> > "fetch\_time": "159ms",  
> > "fetch\_time\_in\_millis": 159,  
> > "fetch\_current": 0  
> > }  
> > },  
> > "process": {  
> > "timestamp": 1378366972506,  
> > "open\_file\_descriptors": 1004,  
> > "cpu": {  
> > "percent": 2,  
> > "sys": "34 seconds and 370 milliseconds",  
> > "sys\_in\_millis": 34370,  
> > "user": "5 minutes, 28 seconds and 710 milliseconds",  
> > "user\_in\_millis": 328710,  
> > "total": "6 minutes, 3 seconds and 80 milliseconds",  
> > "total\_in\_millis": 363080  
> > },  
> > "mem": {  
> > "resident": "397.2mb",  
> > "resident\_in\_bytes": 416587776,  
> > "share": "10.2mb",  
> > "share\_in\_bytes": 10788864,  
> > "total\_virtual": "4.3gb",  
> > "total\_virtual\_in\_bytes": 4688175104  
> > }  
> > }  
> > }  
> > }  
> > }
> > 
> > On Thu, Sep 5, 2013 at 12:44 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:  
> > Check with curl localhost:9200/\_cluster/nodes/stats?process the current number of open files.
> > 
> > Jörg
> > 
> > --  
> > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.  
> > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe).  
> > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > 
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [September 5, 2013, 8:28am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/11 "2013-09-05T08:28:52Z")

</div>

During migration, the segments were rebuilt behind the scenes, with  
deletions and inserts during renumbering.

The current file descriptor usage is low, and during the migration it was  
certainly higher.

Please recheck if ulimit setting of 65535 is really active in the current  
process. You need to set it in /etc/security.d/limits.conf and not only in  
ulimit, and a full re-login of the ES user and re-start of the ES process.

Jörg

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Kailashnath\_Reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailashnath_reddy/32/2124_2.png) [@Kailashnath\_Reddy](https://discuss.elastic.co/u/Kailashnath_Reddy)\
**Post date:** [September 5, 2013, 9:18am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/12 "2013-09-05T09:18:09Z")

</div>

Hi Jorg,  
I did set the limits in limits.conf very early on:

- 

```
           hard nofile 70000

```

- 

```
           soft nofile 70000

```

/K

On Thu, Sep 5, 2013 at 1:58 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \<[joergprante@gmail.com](mailto:joergprante@gmail.com)

> wrote:

> During migration, the segments were rebuilt behind the scenes, with  
> deletions and inserts during renumbering.
> 
> The current file descriptor usage is low, and during the migration it was  
> certainly higher.
> 
> Please recheck if ulimit setting of 65535 is really active in the current  
> process. You need to set it in /etc/security.d/limits.conf and not only in  
> ulimit, and a full re-login of the ES user and re-start of the ES process.
> 
> Jörg
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit  
> [https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [September 5, 2013, 2:03pm UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/13 "2013-09-05T14:03:41Z")

</div>

The "nofile" setting with an asterisk (\*) may not work (you set the default  
value only). The Elasticsearch user must be set in limits.conf.

Jörg

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Kailashnath\_Reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailashnath_reddy/32/2124_2.png) [@Kailashnath\_Reddy](https://discuss.elastic.co/u/Kailashnath_Reddy)\
**Post date:** [September 8, 2013, 8:47am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/14 "2013-09-08T08:47:46Z")

</div>

Sorry for getting back late, sure will try that Jörg, thanks!

On Thu, Sep 5, 2013 at 7:33 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \<[joergprante@gmail.com](mailto:joergprante@gmail.com)

> wrote:

> The "nofile" setting with an asterisk (\*) may not work (you set the  
> default value only). The Elasticsearch user must be set in limits.conf.
> 
> Jörg
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit  
> [https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:17am UTC](https://discuss.elastic.co/t/strange-no-of-file-descriptors-eating-up/13468/15 "2017-07-06T02:17:36Z")

</div>