# Elastic to many open files
**URL:** https://discuss.elastic.co/t/elastic-to-many-open-files/247616
**Category:** Elasticsearch
**Created:** [September 5, 2020, 6:44am UTC](https://discuss.elastic.co/t/elastic-to-many-open-files/247616 "2020-09-05T06:44:49Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@akshay\_dindure1994](https://discuss.elastic.co/u/akshay_dindure1994)
#### Post date: [September 5, 2020, 6:44am UTC](https://discuss.elastic.co/t/elastic-to-many-open-files/247616/1 "2020-09-05T06:44:49Z")
Hello Team
I am facing some serious problems in elastic. I am getting below error in elastic logs
/opt/SOME\_DIRECTORY\_NAME/elasticsearch/data/nodes/0/indices/7QRpS9mnSwCHPdBwaLUJag/3/translog/translog-53.ckp: Too many open files
Due to the above error, Elastic is not getting initialized .
With lsof command more the 40000000 files are getting opened.
And when I stop the elastic process only 2k to 4k files are opened.
When elastic process is launched many of the files are getting opened multiple times
for eg with lsof command below file is getting opened more then 60
to 70 time
/opt/SOME\_DIRECTORY\_NAME/elasticsearch/data/nodes/0/indices/zmE65g1QRcCJ6rQIp97w8g/0/translog/translog-107.ckp
This is happening for all the files under this directory:
/opt/SOME\_DIRECTORY\_NAME/elasticsearch/data/nodes/0/indices/
ulimit -a output
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63457
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1000000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
one strange thing which I have noticed is more no of the file are created in the below directory:
/opt/SOME\_DIRECTORY\_NAME/elasticsearch/data/nodes/0/
in this directory, all elastic index data is stored.
More then 400000 file are created in this directory
find /opt/SOME\_DIRECTORY\_NAME/elasticsearch/data/nodes/0/ -type f| wc -l
400000
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 5, 2020, 4:47pm UTC](https://discuss.elastic.co/t/elastic-to-many-open-files/247616/2 "2020-09-05T16:47:44Z")
> [@akshay\_dindure1994](#):
>
> With lsof command more the 40000000 files are getting opened.
`lsof` isn't a reliable way to determine the number of open files: in a multi-threaded application like Elasticsearch it shows an entry per open file _per thread_ which multiplies the true number by a large factor. The actual figure is available from `GET _nodes/stats/fs`.
Can you share the output of `GET _cluster/health` for your cluster please?
---
### Author:  [@akshay\_dindure1994](https://discuss.elastic.co/u/akshay_dindure1994)
#### Post date: [September 5, 2020, 6:12pm UTC](https://discuss.elastic.co/t/elastic-to-many-open-files/247616/3 "2020-09-05T18:12:11Z")
hello sir please find the output for the `GET \_cluster/health command
{
"cluster\_name" : "data-cluster",
"status" : "yellow",
"timed\_out" : false,
"number\_of\_nodes" : 1,
"number\_of\_data\_nodes" : 1,
"active\_primary\_shards" : 2723,
"active\_shards" : 2723,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 2720,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 0,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 0,
"active\_shards\_percent\_as\_number" : 50.02755833180231
}
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 5, 2020, 7:03pm UTC](https://discuss.elastic.co/t/elastic-to-many-open-files/247616/4 "2020-09-05T19:03:07Z")
That's far too many shards for a single node. The manual contains [instructions for reducing your shard count](https://www.elastic.co/guide/en/elasticsearch/reference/current/avoid-oversharding.html#reduce-shard-counts-increase-shard-size).
---
### Author:  [@akshay\_dindure1994](https://discuss.elastic.co/u/akshay_dindure1994)
#### Post date: [September 10, 2020, 2:04pm UTC](https://discuss.elastic.co/t/elastic-to-many-open-files/247616/5 "2020-09-10T14:04:36Z")
Thx @DavidTurner
But now I am trying to recover old data. And there are more the 2000 index created.
Now ,I have set no\_of\_shard=1 and no\_of\_replicas=0
and I am trying to reindex all the data with reindexing using the below script.
```
#!/bin/bash
variable=`curl -k -XGET -u username:password
https://localhost:9200/_cat/indices/akshay* | awk '{print $3}'`
echo $variable
for i in $variable
do
action=`curl -XPOST -u username:password "https://localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d"{\"source\":{\"index\": \"${i}\"},\"dest\": {\"index\": \"${i}_new\"}}"`
curl -XDELETE -u username:password https://localhost:9200/${i} --insecure
action1=`curl -XPOST -u username:password "https://localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d"{\"source\":{\"index\": \"${i}_new\"},\"dest\": {\"index\": \"${i}\"}}"`
curl -XDELETE -u username:password https://localhost:9200/${i}_new
done
```
The script is working as expected when the index size is big but when index size is small all the index with small size are getting deleted
getting below error for
```
action1=`curl -XPOST -u username:password "https://localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d"{\"source\":{\"index\": \"${i}_new\"},\"dest\": {\"index\": \"${i}\"}}"`
```
{"error":{"root\_cause":[{"type":"index\_not\_found\_exception","reason":"no such index","resource.type":"index\_or\_alias","resource.id":"akshay-dindure1\_n\_new","index\_uuid":"_na_","index":"akshay-dindure1\_n\_new"}],"type":"index\_not\_found\_exception","reason":"no such index","resource.type":"index\_or\_alias","resource.id":"akshay-dindure1\_n\_new","index\_uuid":"_na_","index":"akshay-dindure1\_n\_new"},"status":404}
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 10, 2020, 4:27pm UTC](https://discuss.elastic.co/t/elastic-to-many-open-files/247616/6 "2020-09-10T16:27:07Z")
Seems pretty risky to delete the old index without checking the reindex process succeeded first.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 9, 2020, 5:52am UTC](https://discuss.elastic.co/t/elastic-to-many-open-files/247616/8 "2020-10-09T05:52:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.