# Elasticsearch problems **URL:** \ **Category:** Elasticsearch\ **Created:** [September 1, 2015, 6:02am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404 "2015-09-01T06:02:18Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [September 1, 2015, 6:02am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/1 "2015-09-01T06:02:18Z")
over the weekend, i had a couple of problem elasticseach, and wasn't able to recover yet. I only have one elasticsearch node, and there are two time-based indices (e.g. test-2015.09.01) feeding data in. the first error elasticsearch hit is "too many files" checked the elasticsearch startup file in /etc/init.d/ folder the max\_file\_open is set to 65535, but when i run curl -XGET '[http://localhost:9200/\_nodes?process=true&pretty=true](http://localhost:9200/_nodes?process=true&pretty=true)', the max\_file\_descriptor is 4096. my questions are: 1. what is the "too many files" error? why are there so many files open? 2. How to decide a proper value for the MAX\_OPEN\_FILE settings? 3. Is the MAX\_FILE\_OPEN and max\_file\_descriptor referring to the same thing? if yes, why the setting in the startup script doesn't take effect? * * * [test-2015.08.30][0] failed to execute bulk item (index) index {[test-2015.08.30]... org.elasticsearch.index.engine.CreateFailedEngineException: [test-2015.08.30][0] Create failed for ... at org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:275) at org.elasticsearch.index.shard.IndexShard.create(IndexShard.java:483) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:423) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: /var/data/elasticsearch/esearch02/nodes/0/indices/test-2015.08.30/0/index/\_4u5.fdt (Too many open files) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:221) at java.io.FileOutputStream.(FileOutputStream.java:171) at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:384) at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:277) at org.apache.lucene.store.FileSwitchDirectory.createOutput(FileSwitchDirectory.java:152) at org.apache.lucene.store.RateLimitedFSDirectory.createOutput(RateLimitedFSDirectory.java:40) at org.apache.lucene.store.FilterDirectory.createOutput(FilterDirectory.java:69) at org.apache.lucene.store.FilterDirectory.createOutput(FilterDirectory.java:69) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:113) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:83) at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:270) at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:314) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:241) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:465) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1526) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1252) at org.elasticsearch.index.engine.InternalEngine.innerCreateNoLock(InternalEngine.java:356) at org.elasticsearch.index.engine.InternalEngine.innerCreate(InternalEngine.java:298) at org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:269) I restarted the all services: redis, indexers, and elasticsearch, but seems it didn't recover: elasticsearch log: [2015-09-01 11:13:56,437][DEBUG][action.search.type] [xxxx] All shards failed for phase: [query\_fetch] org.elasticsearch.action.NoShardAvailableActionException: [.marvel-2015.09.01][0] null at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160) at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:57) at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:47) ... ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 1, 2015, 6:15am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/2 "2015-09-01T06:15:27Z")
Most likely because you have too many shards per node. Try to check that as well (at first?)... ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 1, 2015, 6:18am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/3 "2015-09-01T06:18:14Z")
You don't want to set ulimit in that file. Take a look at this chapter of the Definitive Guide - [https://www.elastic.co/guide/en/elasticsearch/guide/current/deploy.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/deploy.html) ---
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [September 1, 2015, 9:40am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/4 "2015-09-01T09:40:00Z")
hmm.. i actually never bothered about the shards... I have 86 shards: {"cluster\_name":"esearch02","status":"yellow","timed\_out":false,"number\_of\_nodes":1,"number\_of\_data\_nodes":1,"active\_primary\_shards":86,"active\_shards":86,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":86,"number\_of\_pending\_tasks":0,"number\_of\_in\_flight\_fetch":0} And from head: cluster health: yellow (86 of 172) is there a way to control the number of shards? and what's the optimal setting? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 1, 2015, 11:24am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/5 "2015-09-01T11:24:08Z")
You set shard count when you create the index, you can control that via a template or other methods. There's no optimal setting, it depends on your use case. ---
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [September 2, 2015, 6:36am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/6 "2015-09-02T06:36:40Z")
I used all the default setting, i.e. 5 primary shards per index and 1 replica, looks like a lot to do to decide the settings... Back to my original question, could you help to explain a little bit more about this "too many files" problem? what is this error? why are there so many files open? Thank you. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 2, 2015, 7:21am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/7 "2015-09-02T07:21:13Z")
Because the underlying lucene engine creates a bunch of files to manage the shard, plus factor in open ports and what not and it adds up quickly. ---
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [September 2, 2015, 11:07am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/8 "2015-09-02T11:07:32Z")
sounds complicated, thank you very much! ---
**Author:** ![amits](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@amits](https://discuss.elastic.co/u/amits)\ **Post date:** [June 13, 2017, 1:21pm UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/9 "2017-06-13T13:21:53Z")
hi @zpp I am also getting the same problem in my ES. Can you please tell me how to increase the no. of shards in order to get rid of this **noshardavailableactionexception** ERROR. thanks in advance. amits ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 13, 2017, 10:54pm UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/10 "2017-06-13T22:54:41Z")
Please start another thread for your question. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 4:05am UTC](https://discuss.elastic.co/t/elasticsearch-problems/28404/11 "2022-11-04T04:05:02Z")