# Elasticsearch Status Red - Help Required
**URL:** https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875
**Category:** Elasticsearch
**Created:** [August 1, 2016, 9:43am UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875 "2016-08-01T09:43:48Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@uttam.maji](https://discuss.elastic.co/u/uttam.maji)
#### Post date: [August 1, 2016, 9:43am UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/1 "2016-08-01T09:43:48Z")
Hi Team,
After creating certain number of index in Elastic Search (ES), when I am trying to restart the server it is not able to recover earlier created indices.
Hence ES web URL shows Status is RED and its not running.
Earlier I changed the data directory and created the indices again, then it ran for few indices but now I am facing the same issue again.
Hence its very difficult to create all the indices again by changing the data directory.
Could you please help me here, if there is any way to resolve the issue without changing the data directory and without recreating the indices ?
Thanks & Regards
Uttam Maji
---
### Author:  [@Ravi\_Shanker\_Reddy](https://discuss.elastic.co/u/Ravi_Shanker_Reddy)
#### Post date: [August 1, 2016, 10:39am UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/2 "2016-08-01T10:39:20Z")
Can you please share your cluster details (How many nodes) , ES version and
also logs.
---
### Author:  [@uttam.maji](https://discuss.elastic.co/u/uttam.maji)
#### Post date: [August 1, 2016, 11:00am UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/3 "2016-08-01T11:00:19Z")
Hi Ravi,
Please find details below
1. single node
2. ES version 2.2.0
3. ES Log file details
[87]: index [.marvel-es-2016.07.28], type [shards], id [4h43RPMqRNat2B\_z5DKfTQ:sLjUqtgoQmSPmUca4bSgnA:agl\_cb\_sd-ade0-ab00\_disk\_space:2:p], message [NodeClosedException[node closed {Tyrannus}{sLjUqtgoQmSPmUca4bSgnA}{127.0.0.1}{127.0.0.1:9300}]]
Caused by: java.nio.file.FileSystemException: /installdir/ELK/elasticsearch-2.2.0/data1/elasticsearch/nodes/0/indices/agl\_cb\_sd-3461-1598\_disk\_space/4/translog/translog.ckp: Too many open files
[2016-07-28 04:45:29,056][WARN][cluster.action.shard] [Tyrannus] [cb\_jpt2\_disk\_space][2] received shard failed for [cb\_jpt2\_disk\_space][2], node[sLjUqtgoQmSPmUca4bSgnA], [P], v[33], s[INITIALIZING], a[id=8-MyaId0SmS7U7dXwka7aA], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2016-07-28T08:45:28.781Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: FileSystemException[/installdir/ELK/elasticsearch-2.2.0/data1/elasticsearch/nodes/0/indices/cb\_jpt2\_disk\_space/2/index/\_a\_Lucene50\_0.tim: Too many open files]; ]], indexUUID [lJMKGd\_wSFCW1b\_WyZlRng], message [master {Tyrannus}{sLjUqtgoQmSPmUca4bSgnA}{127.0.0.1}{127.0.0.1:9300} marked shard as initializing, but shard is marked as failed, resend shard failure], failure [Unknown]
[2016-07-28 04:45:29,057][INFO][node] [Tyrannus] stopping ...
[2016-07-28 04:45:29,072][WARN][cluster.action.shard] [Tyrannus] [agl\_cb\_1598\_jvm\_status][1] received shard failed for [agl\_cb\_1598\_jvm\_status][1], node[sLjUqtgoQmSPmUca4bSgnA], [P], v[17], s[INITIALIZING], a[id=DAJ3npIsT2aZLodNNluo8w], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2016-07-28T08:45:28.841Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: FileSystemException[/installdir/ELK/elasticsearch-2.2.0/data1/elasticsearch/nodes/0/indices/agl\_cb\_1598\_jvm\_status/1/index: Too many open files]; ]], indexUUID [wBy1o9P6SMiIi5lGx63izw], message [master {Tyrannus}{sLjUqtgoQmSPmUca4bSgnA}{127.0.0.1}{127.0.0.1:9300} marked shard as initializing, but shard is marked as failed, resend shard failure], failure [Unknown]
[2016-07-28 04:45:29,075][WARN][indices.memory] [Tyrannus] failed to set shard [agl\_cb\_sd-3461-1598\_disk\_space][4] index buffer to [4mb]
---
### Author:  [@Ravi\_Shanker\_Reddy](https://discuss.elastic.co/u/Ravi_Shanker_Reddy)
#### Post date: [August 1, 2016, 11:10am UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/4 "2016-08-01T11:10:32Z")
> [@uttam.maji](#):
>
> Caused by: java.nio.file.FileSystemException: /installdir/ELK/elasticsearch-2.2.0/data1/elasticsearch/nodes/0/indices/agl\_cb\_sd-3461-1598\_disk\_space/4/translog/translog.ckp: Too many open files
>
From your logs I think this method will solve. You didnt mention your OS model.
For `Red Hat Enterprise Linux Server release 6.4 (Santiago)`
Open the file
`vim /etc/security/limits.conf`
& write
`\* - nofile 999999
-
```
soft nofile 999999
```
-
```
hard nofile 999999`
```
& then try to bring up your server.
---
### Author:  [@uttam.maji](https://discuss.elastic.co/u/uttam.maji)
#### Post date: [August 1, 2016, 11:41am UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/5 "2016-08-01T11:41:13Z")
OS is Linux.
Tried adding details in /etc/security/limits.conf
But still the same issue. Got below logs in terminal.
[96]: index [.marvel-es-2016.08.01], type [shards], id [s3fYEnEyTfKirOQr57vQQQ:\_na:agl\_cb\_57da\_jvm\_status:1:r], message [NodeClosedException[node closed {Joe Fixit}{iVjAAceVS9SuYtdZhuDjVQ}{127.0.0.1}{127.0.0.1:9300}]]
[97]: index [.marvel-es-2016.08.01], type [shards], id [s3fYEnEyTfKirOQr57vQQQ:\_na:agl\_cb\_57da\_jvm\_status:3:p], message [NodeClosedException[node closed {Joe Fixit}{iVjAAceVS9SuYtdZhuDjVQ}{127.0.0.1}{127.0.0.1:9300}]]
[98]: index [.marvel-es-2016.08.01], type [shards], id [s3fYEnEyTfKirOQr57vQQQ:\_na:agl\_cb\_57da\_jvm\_status:3:r], message [NodeClosedException[node closed {Joe Fixit}{iVjAAceVS9SuYtdZhuDjVQ}{127.0.0.1}{127.0.0.1:9300}]]
[99]: index [.marvel-es-2016.08.01], type [shards], id [s3fYEnEyTfKirOQr57vQQQ:\_na:agl\_cb\_57da\_jvm\_status:4:p], message [NodeClosedException[node closed {Joe Fixit}{iVjAAceVS9SuYtdZhuDjVQ}{127.0.0.1}{127.0.0.1:9300}]]]
at org.elasticsearch.marvel.agent.exporter.local.LocalBulk.flush(LocalBulk.java:114)
at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:101)
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 1, 2016, 12:12pm UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/6 "2016-08-01T12:12:53Z")
You said you created a 'certain number of indices'. How many indices/shards do you have on the node? How much heap do you have assigned to the node?
---
### Author:  [@uttam.maji](https://discuss.elastic.co/u/uttam.maji)
#### Post date: [August 1, 2016, 1:55pm UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/7 "2016-08-01T13:55:23Z")
I have total 117 indices. and using default heap size. How to check this?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 2, 2016, 9:10am UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/8 "2016-08-02T09:10:29Z")
If you are using the default configurations your 117 indices might each consist of 5 shards, which gives over 500 shards. This is a lot for a node with the default heap size go 1GB. You can find out how much heap you have through the [node stats API](https://www.elastic.co/guide/en/elasticsearch/reference/master/cluster-nodes-stats.html), e.g.: `curl -XGET 'http://localhost:9200/_nodes/stats/jvm'`
If you are running with default settings I would recommend reducing the number of indices/shards and/or increasing the heap size.
---
### Author:  [@uttam.maji](https://discuss.elastic.co/u/uttam.maji)
#### Post date: [August 2, 2016, 2:20pm UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/9 "2016-08-02T14:20:49Z")
Hi Christian,
I tried increasing the heap size to 4gb, but still its failing.
Please could you help ?
[2016-08-02 09:24:29,858][INFO][gateway] [Emplate] recovered [118] indices into cluster\_state
[2016-08-02 09:24:35,283][WARN][indices.cluster] [Emplate] [[agl\_cb\_sd-3461-1598\_disk\_space][4]] marking and sending shard failed due to [failed recovery]
[agl\_cb\_sd-3461-1598\_disk\_space][[agl\_cb\_sd-3461-1598\_disk\_space][4]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/installdir/ELK/elasticsearch-2.2.0/data1/elasticsearch/nodes/0/indices/agl\_cb\_sd-3461-1598\_disk\_space/4/translog/translog-6.ckp];
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:254)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: [agl\_cb\_sd-3461-1598\_disk\_space][[agl\_cb\_sd-3461-1598\_disk\_space][4]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/installdir/ELK/elasticsearch-2.2.0/data1/elasticsearch/nodes/0/indices/agl\_cb\_sd-3461-1598\_disk\_space/4/translog/translog-6.ckp];
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:156)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1450)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1434)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:925)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:897)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:245)
... 5 more
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:30pm UTC](https://discuss.elastic.co/t/elasticsearch-status-red-help-required/56875/10 "2017-07-05T22:30:44Z")