# Gradual Growth in the number of open files in linux

**URL:** <https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985>\
**Category:** Elasticsearch\
**Created:** [June 1, 2010, 11:23am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985 "2010-06-01T11:23:12Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![shay\_katz](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shay\_katz](https://discuss.elastic.co/u/shay_katz)\
**Post date:** [June 1, 2010, 11:23am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/1 "2010-06-01T11:23:12Z")

</div>

Hi,  
I'm conducting a test for create and search indexes(5 case creation  
every 1 seconds and 1 search every ~ 2 seconds.  
I monitored the number of open files every 1 minute (lsof | wc -l) and  
it seems there is a gradual growth in the number of open files though  
there is no increasing in the load over the time.  
After ~ 20 hours 17K open files were on the system, 15K of them belong  
to the elastic search.  
Is it a noraml behaior of the elastic search?  
I have a file with the list of the open files and excel which shows  
the trend of the number of open files on the system, so if you would  
like to see it, please send me a mail and i'll send you these files.  
Thanks,  
Shay

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 1, 2010, 5:41pm UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/2 "2010-06-01T17:41:39Z")

</div>

This should not happen. Is it still running? Can you mail me the a sample of  
the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:

> Hi,  
> I'm conducting a test for create and search indexes(5 case creation  
> every 1 seconds and 1 search every ~ 2 seconds.  
> I monitored the number of open files every 1 minute (lsof | wc -l) and  
> it seems there is a gradual growth in the number of open files though  
> there is no increasing in the load over the time.  
> After ~ 20 hours 17K open files were on the system, 15K of them belong  
> to the Elasticsearch.  
> Is it a noraml behaior of the Elasticsearch?  
> I have a file with the list of the open files and excel which shows  
> the trend of the number of open files on the system, so if you would  
> like to see it, please send me a mail and i'll send you these files.  
> Thanks,  
> Shay

---

<div class="post-metadata">

**Author:** ![shay\_katz](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shay\_katz](https://discuss.elastic.co/u/shay_katz)\
**Post date:** [June 2, 2010, 8:16am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/3 "2010-06-02T08:16:32Z")

</div>

Hi,  
I executed a new perfromance test for the Elasticsearch, and have the same  
results.  
Attached please find 2 files:

1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of the  
number of open files (_lsof | wc -l_ is running on the system every 1  
minute).  
btw  
Please look in the following output:  
[root]# \*lsof | wc -l  
\*23917  
[root]# \*lsof | grep elastic | wc -l  
\*23376  
2.The second file (ElasticSearchOpenFiles.zip) contain the output from the  
following command: lsof | grep elastic

Thanks,  
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> This should not happen. Is it still running? Can you mail me the a sample  
> of the lsof output?
> 
> On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> 
> > Hi,  
> > I'm conducting a test for create and search indexes(5 case creation  
> > every 1 seconds and 1 search every ~ 2 seconds.  
> > I monitored the number of open files every 1 minute (lsof | wc -l) and  
> > it seems there is a gradual growth in the number of open files though  
> > there is no increasing in the load over the time.  
> > After ~ 20 hours 17K open files were on the system, 15K of them belong  
> > to the Elasticsearch.  
> > Is it a noraml behaior of the Elasticsearch?  
> > I have a file with the list of the open files and excel which shows  
> > the trend of the number of open files on the system, so if you would  
> > like to see it, please send me a mail and i'll send you these files.  
> > Thanks,  
> > Shay

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 2, 2010, 9:29am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/4 "2010-06-02T09:29:57Z")

</div>

This certainly does not look good. I have just run a test against 0.8 with  
2k TPS index and search operations (simple ones) and the open files system  
seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you explain  
more about the test that you run? Go into details regarding the operations  
that you do. Which search requests, what parameters? Do you use scroll  
search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:

> Hi,  
> I executed a new perfromance test for the Elasticsearch, and have the same  
> results.  
> Attached please find 2 files:
> 
> 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of the  
> number of open files (_lsof | wc -l_ is running on the system every 1  
> minute).  
> btw  
> Please look in the following output:  
> [root]# \*lsof | wc -l  
> \*23917  
> [root]# \*lsof | grep elastic | wc -l  
> \*23376  
> 2.The second file (ElasticSearchOpenFiles.zip) contain the output from the  
> following command: lsof | grep elastic
> 
> Thanks,  
> Shay
> 
> On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > This should not happen. Is it still running? Can you mail me the a sample  
> > of the lsof output?
> > 
> > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > 
> > > Hi,  
> > > I'm conducting a test for create and search indexes(5 case creation  
> > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > I monitored the number of open files every 1 minute (lsof | wc -l) and  
> > > it seems there is a gradual growth in the number of open files though  
> > > there is no increasing in the load over the time.  
> > > After ~ 20 hours 17K open files were on the system, 15K of them belong  
> > > to the Elasticsearch.  
> > > Is it a noraml behaior of the Elasticsearch?  
> > > I have a file with the list of the open files and excel which shows  
> > > the trend of the number of open files on the system, so if you would  
> > > like to see it, please send me a mail and i'll send you these files.  
> > > Thanks,  
> > > Shay

---

<div class="post-metadata">

**Author:** ![shay\_katz](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shay\_katz](https://discuss.elastic.co/u/shay_katz)\
**Post date:** [June 3, 2010, 6:37am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/5 "2010-06-03T06:37:40Z")

</div>

The test was executed with load runner.  
I'll try to describe it.  
You can see in the attached file(create\_case.txt) the http call for _create  
cases_.  
Parameters description:

- {UserId} - we have 20 concurrent users each of them have a unique  
{UserId}
- {objectId} - number which increasing every iteration in 1
- {Description1}- we have a file with ~ 70K lines so in each iteration  
the Load Runner choose 1 random line for description
- {title} - the same as description - but another file for title
- {Status} - rnadom staus (overall there are 3 statuses
- {First\_RndNumber} - random nuber between 1 to 800
- The all other parameters contain random number

_Search cases:_  
you can see in the attached file(SearchCases.txt) the search operation,  
please let me know if you need clarification for this action.

_Workload:_  
Create cases was executed with 20 concurrent users which create case every ~  
4 seconds  
Search cases was executed with 10 concurrent users which executed the search  
every ~25 seconds.  
The create and the search run on the same time on the system.

Thanks,  
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> This certainly does not look good. I have just run a test against 0.8 with  
> 2k TPS index and search operations (simple ones) and the open files system  
> seems to be good (constant around 80) (with an FS gateway).
> 
> Maybe you uncovered a problem here that I can't recreate. Can you explain  
> more about the test that you run? Go into details regarding the operations  
> that you do. Which search requests, what parameters? Do you use scroll  
> search request?
> 
> Maybe you can create a simple test case that I can run here?
> 
> -shay.banon
> 
> On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> 
> > Hi,  
> > I executed a new perfromance test for the Elasticsearch, and have the  
> > same results.  
> > Attached please find 2 files:
> > 
> > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of the  
> > number of open files (_lsof | wc -l_ is running on the system every 1  
> > minute).  
> > btw  
> > Please look in the following output:  
> > [root]# \*lsof | wc -l  
> > \*23917  
> > [root]# \*lsof | grep elastic | wc -l  
> > \*23376  
> > 2.The second file (ElasticSearchOpenFiles.zip) contain the output from the  
> > following command: lsof | grep elastic
> > 
> > Thanks,  
> > Shay
> > 
> > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > 
> > > This should not happen. Is it still running? Can you mail me the a sample  
> > > of the lsof output?
> > > 
> > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > 
> > > > Hi,  
> > > > I'm conducting a test for create and search indexes(5 case creation  
> > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > I monitored the number of open files every 1 minute (lsof | wc -l) and  
> > > > it seems there is a gradual growth in the number of open files though  
> > > > there is no increasing in the load over the time.  
> > > > After ~ 20 hours 17K open files were on the system, 15K of them belong  
> > > > to the Elasticsearch.  
> > > > Is it a noraml behaior of the Elasticsearch?  
> > > > I have a file with the list of the open files and excel which shows  
> > > > the trend of the number of open files on the system, so if you would  
> > > > like to see it, please send me a mail and i'll send you these files.  
> > > > Thanks,  
> > > > Shay

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 3, 2010, 9:44am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/6 "2010-06-03T09:44:01Z")

</div>

ok. I will try and run something like this, but, if you want to move things  
faster, it would help if you can create a reproduction that I can run. Also,  
how many nodes are running? Can you share you configuration? Do you create  
an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:

> The test was executed with load runner.  
> I'll try to describe it.  
> You can see in the attached file(create\_case.txt) the http call for _create  
> cases_.  
> Parameters description:
> 
> - {UserId} - we have 20 concurrent users each of them have a unique  
> {UserId}
> - {objectId} - number which increasing every iteration in 1
> - {Description1}- we have a file with ~ 70K lines so in each iteration  
> the Load Runner choose 1 random line for description
> - {title} - the same as description - but another file for title
> - {Status} - rnadom staus (overall there are 3 statuses
> - {First\_RndNumber} - random nuber between 1 to 800
> - The all other parameters contain random number
> 
> _Search cases:_  
> you can see in the attached file(SearchCases.txt) the search operation,  
> please let me know if you need clarification for this action.
> 
> _Workload:_  
> Create cases was executed with 20 concurrent users which create case every  
> ~ 4 seconds  
> Search cases was executed with 10 concurrent users which executed the  
> search every ~25 seconds.  
> The create and the search run on the same time on the system.
> 
> Thanks,  
> Shay
> 
> On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > This certainly does not look good. I have just run a test against 0.8 with  
> > 2k TPS index and search operations (simple ones) and the open files system  
> > seems to be good (constant around 80) (with an FS gateway).
> > 
> > Maybe you uncovered a problem here that I can't recreate. Can you explain  
> > more about the test that you run? Go into details regarding the operations  
> > that you do. Which search requests, what parameters? Do you use scroll  
> > search request?
> > 
> > Maybe you can create a simple test case that I can run here?
> > 
> > -shay.banon
> > 
> > On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > 
> > > Hi,  
> > > I executed a new perfromance test for the Elasticsearch, and have the  
> > > same results.  
> > > Attached please find 2 files:
> > > 
> > > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of the  
> > > number of open files (_lsof | wc -l_ is running on the system every 1  
> > > minute).  
> > > btw  
> > > Please look in the following output:  
> > > [root]# \*lsof | wc -l  
> > > \*23917  
> > > [root]# \*lsof | grep elastic | wc -l  
> > > \*23376  
> > > 2.The second file (ElasticSearchOpenFiles.zip) contain the output from  
> > > the following command: lsof | grep elastic
> > > 
> > > Thanks,  
> > > Shay
> > > 
> > > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > 
> > > > This should not happen. Is it still running? Can you mail me the a  
> > > > sample of the lsof output?
> > > > 
> > > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > 
> > > > > Hi,  
> > > > > I'm conducting a test for create and search indexes(5 case creation  
> > > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > > I monitored the number of open files every 1 minute (lsof | wc -l) and  
> > > > > it seems there is a gradual growth in the number of open files though  
> > > > > there is no increasing in the load over the time.  
> > > > > After ~ 20 hours 17K open files were on the system, 15K of them belong  
> > > > > to the Elasticsearch.  
> > > > > Is it a noraml behaior of the Elasticsearch?  
> > > > > I have a file with the list of the open files and excel which shows  
> > > > > the trend of the number of open files on the system, so if you would  
> > > > > like to see it, please send me a mail and i'll send you these files.  
> > > > > Thanks,  
> > > > > Shay

---

<div class="post-metadata">

**Author:** ![shay\_katz](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shay\_katz](https://discuss.elastic.co/u/shay_katz)\
**Post date:** [June 3, 2010, 10:53am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/7 "2010-06-03T10:53:02Z")

</div>

Attached is the yml file.  
I'm running on a single node.  
I have 20 users for each one of them i created an index (20 indexes in the  
total)  
btw,  
optimize+flush were executed every 15 minutes  
Thanks,  
Shay

On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> ok. I will try and run something like this, but, if you want to move things  
> faster, it would help if you can create a reproduction that I can run. Also,  
> how many nodes are running? Can you share you configuration? Do you create  
> an index per user?
> 
> -shay.banon
> 
> On Thu, Jun 3, 2010 at 9:37 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> 
> > The test was executed with load runner.  
> > I'll try to describe it.  
> > You can see in the attached file(create\_case.txt) the http call for _create  
> > cases_.  
> > Parameters description:
> > 
> > - {UserId} - we have 20 concurrent users each of them have a unique  
> > {UserId}
> > - {objectId} - number which increasing every iteration in 1
> > - {Description1}- we have a file with ~ 70K lines so in each iteration  
> > the Load Runner choose 1 random line for description
> > - {title} - the same as description - but another file for title
> > - {Status} - rnadom staus (overall there are 3 statuses
> > - {First\_RndNumber} - random nuber between 1 to 800
> > - The all other parameters contain random number
> > 
> > _Search cases:_  
> > you can see in the attached file(SearchCases.txt) the search operation,  
> > please let me know if you need clarification for this action.
> > 
> > _Workload:_  
> > Create cases was executed with 20 concurrent users which create case every  
> > ~ 4 seconds  
> > Search cases was executed with 10 concurrent users which executed the  
> > search every ~25 seconds.  
> > The create and the search run on the same time on the system.
> > 
> > Thanks,  
> > Shay
> > 
> > On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > 
> > > wrote:
> > 
> > > This certainly does not look good. I have just run a test against 0.8  
> > > with 2k TPS index and search operations (simple ones) and the open files  
> > > system seems to be good (constant around 80) (with an FS gateway).
> > > 
> > > Maybe you uncovered a problem here that I can't recreate. Can you explain  
> > > more about the test that you run? Go into details regarding the operations  
> > > that you do. Which search requests, what parameters? Do you use scroll  
> > > search request?
> > > 
> > > Maybe you can create a simple test case that I can run here?
> > > 
> > > -shay.banon
> > > 
> > > On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > 
> > > > Hi,  
> > > > I executed a new perfromance test for the Elasticsearch, and have the  
> > > > same results.  
> > > > Attached please find 2 files:
> > > > 
> > > > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of the  
> > > > number of open files (_lsof | wc -l_ is running on the system every 1  
> > > > minute).  
> > > > btw  
> > > > Please look in the following output:  
> > > > [root]# \*lsof | wc -l  
> > > > \*23917  
> > > > [root]# \*lsof | grep elastic | wc -l  
> > > > \*23376  
> > > > 2.The second file (ElasticSearchOpenFiles.zip) contain the output from  
> > > > the following command: lsof | grep elastic
> > > > 
> > > > Thanks,  
> > > > Shay
> > > > 
> > > > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > 
> > > > > This should not happen. Is it still running? Can you mail me the a  
> > > > > sample of the lsof output?
> > > > > 
> > > > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > > 
> > > > > > Hi,  
> > > > > > I'm conducting a test for create and search indexes(5 case creation  
> > > > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > > > I monitored the number of open files every 1 minute (lsof | wc -l) and  
> > > > > > it seems there is a gradual growth in the number of open files though  
> > > > > > there is no increasing in the load over the time.  
> > > > > > After ~ 20 hours 17K open files were on the system, 15K of them belong  
> > > > > > to the Elasticsearch.  
> > > > > > Is it a noraml behaior of the Elasticsearch?  
> > > > > > I have a file with the list of the open files and excel which shows  
> > > > > > the trend of the number of open files on the system, so if you would  
> > > > > > like to see it, please send me a mail and i'll send you these files.  
> > > > > > Thanks,  
> > > > > > Shay

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 3, 2010, 11:04am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/8 "2010-06-03T11:04:11Z")

</div>

One more thing, any special configuration when creating an index? Also, the  
big difference, I think from what I did to what you did is running optimize,  
can you try without it and see if it still happens? You don't really need to  
call either.

On Thu, Jun 3, 2010 at 1:53 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:

> Attached is the yml file.  
> I'm running on a single node.  
> I have 20 users for each one of them i created an index (20 indexes in the  
> total)  
> btw,  
> optimize+flush were executed every 15 minutes  
> Thanks,  
> Shay
> 
> On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > ok. I will try and run something like this, but, if you want to move  
> > things faster, it would help if you can create a reproduction that I can  
> > run. Also, how many nodes are running? Can you share you configuration? Do  
> > you create an index per user?
> > 
> > -shay.banon
> > 
> > On Thu, Jun 3, 2010 at 9:37 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > 
> > > The test was executed with load runner.  
> > > I'll try to describe it.  
> > > You can see in the attached file(create\_case.txt) the http call for _create  
> > > cases_.  
> > > Parameters description:
> > > 
> > > - {UserId} - we have 20 concurrent users each of them have a unique  
> > > {UserId}
> > > - {objectId} - number which increasing every iteration in 1
> > > - {Description1}- we have a file with ~ 70K lines so in  
> > > each iteration the Load Runner choose 1 random line for description
> > > - {title} - the same as description - but another file for title
> > > - {Status} - rnadom staus (overall there are 3 statuses
> > > - {First\_RndNumber} - random nuber between 1 to 800
> > > - The all other parameters contain random number
> > > 
> > > _Search cases:_  
> > > you can see in the attached file(SearchCases.txt) the search operation,  
> > > please let me know if you need clarification for this action.
> > > 
> > > _Workload:_  
> > > Create cases was executed with 20 concurrent users which create case  
> > > every ~ 4 seconds  
> > > Search cases was executed with 10 concurrent users which executed the  
> > > search every ~25 seconds.  
> > > The create and the search run on the same time on the system.
> > > 
> > > Thanks,  
> > > Shay
> > > 
> > > On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon \<  
> > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > 
> > > > This certainly does not look good. I have just run a test against 0.8  
> > > > with 2k TPS index and search operations (simple ones) and the open files  
> > > > system seems to be good (constant around 80) (with an FS gateway).
> > > > 
> > > > Maybe you uncovered a problem here that I can't recreate. Can you  
> > > > explain more about the test that you run? Go into details regarding the  
> > > > operations that you do. Which search requests, what parameters? Do you use  
> > > > scroll search request?
> > > > 
> > > > Maybe you can create a simple test case that I can run here?
> > > > 
> > > > -shay.banon
> > > > 
> > > > On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > 
> > > > > Hi,  
> > > > > I executed a new perfromance test for the Elasticsearch, and have the  
> > > > > same results.  
> > > > > Attached please find 2 files:
> > > > > 
> > > > > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of the  
> > > > > number of open files (_lsof | wc -l_ is running on the system every 1  
> > > > > minute).  
> > > > > btw  
> > > > > Please look in the following output:  
> > > > > [root]# \*lsof | wc -l  
> > > > > \*23917  
> > > > > [root]# \*lsof | grep elastic | wc -l  
> > > > > \*23376  
> > > > > 2.The second file (ElasticSearchOpenFiles.zip) contain the output from  
> > > > > the following command: lsof | grep elastic
> > > > > 
> > > > > Thanks,  
> > > > > Shay
> > > > > 
> > > > > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > 
> > > > > > This should not happen. Is it still running? Can you mail me the a  
> > > > > > sample of the lsof output?
> > > > > > 
> > > > > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > > > 
> > > > > > > Hi,  
> > > > > > > I'm conducting a test for create and search indexes(5 case creation  
> > > > > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > > > > I monitored the number of open files every 1 minute (lsof | wc -l)  
> > > > > > > and  
> > > > > > > it seems there is a gradual growth in the number of open files though  
> > > > > > > there is no increasing in the load over the time.  
> > > > > > > After ~ 20 hours 17K open files were on the system, 15K of them  
> > > > > > > belong  
> > > > > > > to the Elasticsearch.  
> > > > > > > Is it a noraml behaior of the Elasticsearch?  
> > > > > > > I have a file with the list of the open files and excel which shows  
> > > > > > > the trend of the number of open files on the system, so if you would  
> > > > > > > like to see it, please send me a mail and i'll send you these files.  
> > > > > > > Thanks,  
> > > > > > > Shay

---

<div class="post-metadata">

**Author:** ![shay\_katz](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shay\_katz](https://discuss.elastic.co/u/shay_katz)\
**Post date:** [June 3, 2010, 11:15am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/9 "2010-06-03T11:15:51Z")

</div>

I don't think i made any special configuration fore creating the case , it  
looks like:  
http://{ServerName}:{Port}/_{UserId}_/CASE/{UserId}{objectId}{UserIdhttp://{ServerName}:{Port}/{UserId}/CASE/{UserId}{objectId}{UserId  
}  
when {UserId} is a constant number for each user.  
I executed the optimize after performance test without it which shows the  
same phenomenon.

On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> One more thing, any special configuration when creating an index? Also, the  
> big difference, I think from what I did to what you did is running optimize,  
> can you try without it and see if it still happens? You don't really need to  
> call either.
> 
> On Thu, Jun 3, 2010 at 1:53 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> 
> > Attached is the yml file.  
> > I'm running on a single node.  
> > I have 20 users for each one of them i created an index (20 indexes in the  
> > total)  
> > btw,  
> > optimize+flush were executed every 15 minutes  
> > Thanks,  
> > Shay
> > 
> > On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon \<  
> > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > 
> > > ok. I will try and run something like this, but, if you want to move  
> > > things faster, it would help if you can create a reproduction that I can  
> > > run. Also, how many nodes are running? Can you share you configuration? Do  
> > > you create an index per user?
> > > 
> > > -shay.banon
> > > 
> > > On Thu, Jun 3, 2010 at 9:37 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > 
> > > > The test was executed with load runner.  
> > > > I'll try to describe it.  
> > > > You can see in the attached file(create\_case.txt) the http call for _create  
> > > > cases_.  
> > > > Parameters description:
> > > > 
> > > > - {UserId} - we have 20 concurrent users each of them have a unique  
> > > > {UserId}
> > > > - {objectId} - number which increasing every iteration in 1
> > > > - {Description1}- we have a file with ~ 70K lines so in  
> > > > each iteration the Load Runner choose 1 random line for description
> > > > - {title} - the same as description - but another file for title
> > > > - {Status} - rnadom staus (overall there are 3 statuses
> > > > - {First\_RndNumber} - random nuber between 1 to 800
> > > > - The all other parameters contain random number
> > > > 
> > > > _Search cases:_  
> > > > you can see in the attached file(SearchCases.txt) the search operation,  
> > > > please let me know if you need clarification for this action.
> > > > 
> > > > _Workload:_  
> > > > Create cases was executed with 20 concurrent users which create case  
> > > > every ~ 4 seconds  
> > > > Search cases was executed with 10 concurrent users which executed the  
> > > > search every ~25 seconds.  
> > > > The create and the search run on the same time on the system.
> > > > 
> > > > Thanks,  
> > > > Shay
> > > > 
> > > > On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon \<  
> > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > 
> > > > > This certainly does not look good. I have just run a test against 0.8  
> > > > > with 2k TPS index and search operations (simple ones) and the open files  
> > > > > system seems to be good (constant around 80) (with an FS gateway).
> > > > > 
> > > > > Maybe you uncovered a problem here that I can't recreate. Can you  
> > > > > explain more about the test that you run? Go into details regarding the  
> > > > > operations that you do. Which search requests, what parameters? Do you use  
> > > > > scroll search request?
> > > > > 
> > > > > Maybe you can create a simple test case that I can run here?
> > > > > 
> > > > > -shay.banon
> > > > > 
> > > > > On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > 
> > > > > > Hi,  
> > > > > > I executed a new perfromance test for the Elasticsearch, and have the  
> > > > > > same results.  
> > > > > > Attached please find 2 files:
> > > > > > 
> > > > > > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of  
> > > > > > the number of open files (_lsof | wc -l_ is running on the system  
> > > > > > every 1 minute).  
> > > > > > btw  
> > > > > > Please look in the following output:  
> > > > > > [root]# \*lsof | wc -l  
> > > > > > \*23917  
> > > > > > [root]# \*lsof | grep elastic | wc -l  
> > > > > > \*23376  
> > > > > > 2.The second file (ElasticSearchOpenFiles.zip) contain the output from  
> > > > > > the following command: lsof | grep elastic
> > > > > > 
> > > > > > Thanks,  
> > > > > > Shay
> > > > > > 
> > > > > > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > 
> > > > > > > This should not happen. Is it still running? Can you mail me the a  
> > > > > > > sample of the lsof output?
> > > > > > > 
> > > > > > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > > > > 
> > > > > > > > Hi,  
> > > > > > > > I'm conducting a test for create and search indexes(5 case creation  
> > > > > > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > > > > > I monitored the number of open files every 1 minute (lsof | wc -l)  
> > > > > > > > and  
> > > > > > > > it seems there is a gradual growth in the number of open files  
> > > > > > > > though  
> > > > > > > > there is no increasing in the load over the time.  
> > > > > > > > After ~ 20 hours 17K open files were on the system, 15K of them  
> > > > > > > > belong  
> > > > > > > > to the Elasticsearch.  
> > > > > > > > Is it a noraml behaior of the Elasticsearch?  
> > > > > > > > I have a file with the list of the open files and excel which shows  
> > > > > > > > the trend of the number of open files on the system, so if you would  
> > > > > > > > like to see it, please send me a mail and i'll send you these files.  
> > > > > > > > Thanks,  
> > > > > > > > Shay

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 4, 2010, 6:17pm UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/10 "2010-06-04T18:17:26Z")

</div>

Is there a chance that you can create a simple test case that I can run that  
simulates this, it would really help to speed things up. Something that even  
uses bash script with curls, or simple Java program, or something similar?

On Thu, Jun 3, 2010 at 2:15 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:

> I don't think i made any special configuration fore creating the case , it  
> looks like:  
> http://{ServerName}:{Port}/_{UserId}_/CASE/{UserId}{objectId}{UserIdhttp://{ServerName}%3A{Port}/{UserId}/CASE/{UserId}{objectId}{UserId  
> }  
> when {UserId} is a constant number for each user.  
> I executed the optimize after performance test without it which shows the  
> same phenomenon.
> 
> On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > One more thing, any special configuration when creating an index? Also,  
> > the big difference, I think from what I did to what you did is running  
> > optimize, can you try without it and see if it still happens? You don't  
> > really need to call either.
> > 
> > On Thu, Jun 3, 2010 at 1:53 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > 
> > > Attached is the yml file.  
> > > I'm running on a single node.  
> > > I have 20 users for each one of them i created an index (20 indexes in  
> > > the total)  
> > > btw,  
> > > optimize+flush were executed every 15 minutes  
> > > Thanks,  
> > > Shay
> > > 
> > > On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon \<  
> > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > 
> > > > ok. I will try and run something like this, but, if you want to move  
> > > > things faster, it would help if you can create a reproduction that I can  
> > > > run. Also, how many nodes are running? Can you share you configuration? Do  
> > > > you create an index per user?
> > > > 
> > > > -shay.banon
> > > > 
> > > > On Thu, Jun 3, 2010 at 9:37 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > 
> > > > > The test was executed with load runner.  
> > > > > I'll try to describe it.  
> > > > > You can see in the attached file(create\_case.txt) the http call for _create  
> > > > > cases_.  
> > > > > Parameters description:
> > > > > 
> > > > > - {UserId} - we have 20 concurrent users each of them have a unique  
> > > > > {UserId}
> > > > > - {objectId} - number which increasing every iteration in 1
> > > > > - {Description1}- we have a file with ~ 70K lines so in  
> > > > > each iteration the Load Runner choose 1 random line for description
> > > > > - {title} - the same as description - but another file for title
> > > > > - {Status} - rnadom staus (overall there are 3 statuses
> > > > > - {First\_RndNumber} - random nuber between 1 to 800
> > > > > - The all other parameters contain random number
> > > > > 
> > > > > _Search cases:_  
> > > > > you can see in the attached file(SearchCases.txt) the search operation,  
> > > > > please let me know if you need clarification for this action.
> > > > > 
> > > > > _Workload:_  
> > > > > Create cases was executed with 20 concurrent users which create case  
> > > > > every ~ 4 seconds  
> > > > > Search cases was executed with 10 concurrent users which executed the  
> > > > > search every ~25 seconds.  
> > > > > The create and the search run on the same time on the system.
> > > > > 
> > > > > Thanks,  
> > > > > Shay
> > > > > 
> > > > > On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon \<  
> > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > 
> > > > > > This certainly does not look good. I have just run a test against 0.8  
> > > > > > with 2k TPS index and search operations (simple ones) and the open files  
> > > > > > system seems to be good (constant around 80) (with an FS gateway).
> > > > > > 
> > > > > > Maybe you uncovered a problem here that I can't recreate. Can you  
> > > > > > explain more about the test that you run? Go into details regarding the  
> > > > > > operations that you do. Which search requests, what parameters? Do you use  
> > > > > > scroll search request?
> > > > > > 
> > > > > > Maybe you can create a simple test case that I can run here?
> > > > > > 
> > > > > > -shay.banon
> > > > > > 
> > > > > > On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > 
> > > > > > > Hi,  
> > > > > > > I executed a new perfromance test for the Elasticsearch, and have  
> > > > > > > the same results.  
> > > > > > > Attached please find 2 files:
> > > > > > > 
> > > > > > > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of  
> > > > > > > the number of open files (_lsof | wc -l_ is running on the system  
> > > > > > > every 1 minute).  
> > > > > > > btw  
> > > > > > > Please look in the following output:  
> > > > > > > [root]# \*lsof | wc -l  
> > > > > > > \*23917  
> > > > > > > [root]# \*lsof | grep elastic | wc -l  
> > > > > > > \*23376  
> > > > > > > 2.The second file (ElasticSearchOpenFiles.zip) contain the output  
> > > > > > > from the following command: lsof | grep elastic
> > > > > > > 
> > > > > > > Thanks,  
> > > > > > > Shay
> > > > > > > 
> > > > > > > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > 
> > > > > > > > This should not happen. Is it still running? Can you mail me the a  
> > > > > > > > sample of the lsof output?
> > > > > > > > 
> > > > > > > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > > > > > 
> > > > > > > > > Hi,  
> > > > > > > > > I'm conducting a test for create and search indexes(5 case creation  
> > > > > > > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > > > > > > I monitored the number of open files every 1 minute (lsof | wc -l)  
> > > > > > > > > and  
> > > > > > > > > it seems there is a gradual growth in the number of open files  
> > > > > > > > > though  
> > > > > > > > > there is no increasing in the load over the time.  
> > > > > > > > > After ~ 20 hours 17K open files were on the system, 15K of them  
> > > > > > > > > belong  
> > > > > > > > > to the Elasticsearch.  
> > > > > > > > > Is it a noraml behaior of the Elasticsearch?  
> > > > > > > > > I have a file with the list of the open files and excel which shows  
> > > > > > > > > the trend of the number of open files on the system, so if you  
> > > > > > > > > would  
> > > > > > > > > like to see it, please send me a mail and i'll send you these  
> > > > > > > > > files.  
> > > > > > > > > Thanks,  
> > > > > > > > > Shay

---

<div class="post-metadata">

**Author:** ![shay\_katz](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shay\_katz](https://discuss.elastic.co/u/shay_katz)\
**Post date:** [June 5, 2010, 9:19pm UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/11 "2010-06-05T21:19:12Z")

</div>

I can try do this but it'll probably take some time.  
Maybe you have any idea regarding the log level- maybe i can run the same  
test with a different log level which will give me more indecation about the  
cause for many open files?

On Fri, Jun 4, 2010 at 9:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Is there a chance that you can create a simple test case that I can run  
> that simulates this, it would really help to speed things up. Something that  
> even uses bash script with curls, or simple Java program, or something  
> similar?
> 
> On Thu, Jun 3, 2010 at 2:15 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> 
> > I don't think i made any special configuration fore creating the case ,  
> > it looks like:  
> > http://{ServerName}:{Port}/_{UserId}_/CASE/{UserId}{objectId}{UserIdhttp://{servername}%3A{port}/{UserId}/CASE/{UserId}{objectId}{UserId  
> > }  
> > when {UserId} is a constant number for each user.  
> > I executed the optimize after performance test without it which shows the  
> > same phenomenon.
> > 
> > On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> > 
> > > One more thing, any special configuration when creating an index? Also,  
> > > the big difference, I think from what I did to what you did is running  
> > > optimize, can you try without it and see if it still happens? You don't  
> > > really need to call either.
> > > 
> > > On Thu, Jun 3, 2010 at 1:53 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > 
> > > > Attached is the yml file.  
> > > > I'm running on a single node.  
> > > > I have 20 users for each one of them i created an index (20 indexes in  
> > > > the total)  
> > > > btw,  
> > > > optimize+flush were executed every 15 minutes  
> > > > Thanks,  
> > > > Shay
> > > > 
> > > > On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon \<  
> > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > 
> > > > > ok. I will try and run something like this, but, if you want to move  
> > > > > things faster, it would help if you can create a reproduction that I can  
> > > > > run. Also, how many nodes are running? Can you share you configuration? Do  
> > > > > you create an index per user?
> > > > > 
> > > > > -shay.banon
> > > > > 
> > > > > On Thu, Jun 3, 2010 at 9:37 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > > 
> > > > > > The test was executed with load runner.  
> > > > > > I'll try to describe it.  
> > > > > > You can see in the attached file(create\_case.txt) the http call for _create  
> > > > > > cases_.  
> > > > > > Parameters description:
> > > > > > 
> > > > > > - {UserId} - we have 20 concurrent users each of them have a  
> > > > > > unique {UserId}
> > > > > > - {objectId} - number which increasing every iteration in 1
> > > > > > - {Description1}- we have a file with ~ 70K lines so in  
> > > > > > each iteration the Load Runner choose 1 random line for description
> > > > > > - {title} - the same as description - but another file for title
> > > > > > - {Status} - rnadom staus (overall there are 3 statuses
> > > > > > - {First\_RndNumber} - random nuber between 1 to 800
> > > > > > - The all other parameters contain random number
> > > > > > 
> > > > > > _Search cases:_  
> > > > > > you can see in the attached file(SearchCases.txt) the search  
> > > > > > operation, please let me know if you need clarification for this action.
> > > > > > 
> > > > > > _Workload:_  
> > > > > > Create cases was executed with 20 concurrent users which create case  
> > > > > > every ~ 4 seconds  
> > > > > > Search cases was executed with 10 concurrent users which executed the  
> > > > > > search every ~25 seconds.  
> > > > > > The create and the search run on the same time on the system.
> > > > > > 
> > > > > > Thanks,  
> > > > > > Shay
> > > > > > 
> > > > > > On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon \<  
> > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > 
> > > > > > > This certainly does not look good. I have just run a test against 0.8  
> > > > > > > with 2k TPS index and search operations (simple ones) and the open files  
> > > > > > > system seems to be good (constant around 80) (with an FS gateway).
> > > > > > > 
> > > > > > > Maybe you uncovered a problem here that I can't recreate. Can you  
> > > > > > > explain more about the test that you run? Go into details regarding the  
> > > > > > > operations that you do. Which search requests, what parameters? Do you use  
> > > > > > > scroll search request?
> > > > > > > 
> > > > > > > Maybe you can create a simple test case that I can run here?
> > > > > > > 
> > > > > > > -shay.banon
> > > > > > > 
> > > > > > > On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > > 
> > > > > > > > Hi,  
> > > > > > > > I executed a new perfromance test for the Elasticsearch, and have  
> > > > > > > > the same results.  
> > > > > > > > Attached please find 2 files:
> > > > > > > > 
> > > > > > > > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of  
> > > > > > > > the number of open files (_lsof | wc -l_ is running on the system  
> > > > > > > > every 1 minute).  
> > > > > > > > btw  
> > > > > > > > Please look in the following output:  
> > > > > > > > [root]# \*lsof | wc -l  
> > > > > > > > \*23917  
> > > > > > > > [root]# \*lsof | grep elastic | wc -l  
> > > > > > > > \*23376  
> > > > > > > > 2.The second file (ElasticSearchOpenFiles.zip) contain the output  
> > > > > > > > from the following command: lsof | grep elastic
> > > > > > > > 
> > > > > > > > Thanks,  
> > > > > > > > Shay
> > > > > > > > 
> > > > > > > > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > 
> > > > > > > > > This should not happen. Is it still running? Can you mail me the a  
> > > > > > > > > sample of the lsof output?
> > > > > > > > > 
> > > > > > > > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > > > > > > 
> > > > > > > > > > Hi,  
> > > > > > > > > > I'm conducting a test for create and search indexes(5 case  
> > > > > > > > > > creation  
> > > > > > > > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > > > > > > > I monitored the number of open files every 1 minute (lsof | wc -l)  
> > > > > > > > > > and  
> > > > > > > > > > it seems there is a gradual growth in the number of open files  
> > > > > > > > > > though  
> > > > > > > > > > there is no increasing in the load over the time.  
> > > > > > > > > > After ~ 20 hours 17K open files were on the system, 15K of them  
> > > > > > > > > > belong  
> > > > > > > > > > to the Elasticsearch.  
> > > > > > > > > > Is it a noraml behaior of the Elasticsearch?  
> > > > > > > > > > I have a file with the list of the open files and excel which  
> > > > > > > > > > shows  
> > > > > > > > > > the trend of the number of open files on the system, so if you  
> > > > > > > > > > would  
> > > > > > > > > > like to see it, please send me a mail and i'll send you these  
> > > > > > > > > > files.  
> > > > > > > > > > Thanks,  
> > > > > > > > > > Shay

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 6, 2010, 10:37am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/12 "2010-06-06T10:37:42Z")

</div>

Not really, this is usually very low level problem, I need to be able to  
recreate it and basically debug it...

On Sun, Jun 6, 2010 at 12:19 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:

> I can try do this but it'll probably take some time.  
> Maybe you have any idea regarding the log level- maybe i can run the same  
> test with a different log level which will give me more indecation about the  
> cause for many open files?
> 
> On Fri, Jun 4, 2010 at 9:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > Is there a chance that you can create a simple test case that I can run  
> > that simulates this, it would really help to speed things up. Something that  
> > even uses bash script with curls, or simple Java program, or something  
> > similar?
> > 
> > On Thu, Jun 3, 2010 at 2:15 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > 
> > > I don't think i made any special configuration fore creating the case ,  
> > > it looks like:  
> > > http://{ServerName}:{Port}/_{UserId}_/CASE/{UserId}{objectId}{UserIdhttp://{servername}%3A{port}/{UserId}/CASE/{UserId}{objectId}{UserId  
> > > }  
> > > when {UserId} is a constant number for each user.  
> > > I executed the optimize after performance test without it which shows the  
> > > same phenomenon.
> > > 
> > > On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > 
> > > > wrote:
> > > 
> > > > One more thing, any special configuration when creating an index? Also,  
> > > > the big difference, I think from what I did to what you did is running  
> > > > optimize, can you try without it and see if it still happens? You don't  
> > > > really need to call either.
> > > > 
> > > > On Thu, Jun 3, 2010 at 1:53 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > 
> > > > > Attached is the yml file.  
> > > > > I'm running on a single node.  
> > > > > I have 20 users for each one of them i created an index (20 indexes in  
> > > > > the total)  
> > > > > btw,  
> > > > > optimize+flush were executed every 15 minutes  
> > > > > Thanks,  
> > > > > Shay
> > > > > 
> > > > > On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon \<  
> > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > 
> > > > > > ok. I will try and run something like this, but, if you want to move  
> > > > > > things faster, it would help if you can create a reproduction that I can  
> > > > > > run. Also, how many nodes are running? Can you share you configuration? Do  
> > > > > > you create an index per user?
> > > > > > 
> > > > > > -shay.banon
> > > > > > 
> > > > > > On Thu, Jun 3, 2010 at 9:37 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > 
> > > > > > > The test was executed with load runner.  
> > > > > > > I'll try to describe it.  
> > > > > > > You can see in the attached file(create\_case.txt) the http call for  
> > > > > > > _create cases_.  
> > > > > > > Parameters description:
> > > > > > > 
> > > > > > > - {UserId} - we have 20 concurrent users each of them have a  
> > > > > > > unique {UserId}
> > > > > > > - {objectId} - number which increasing every iteration in 1
> > > > > > > - {Description1}- we have a file with ~ 70K lines so in  
> > > > > > > each iteration the Load Runner choose 1 random line for description
> > > > > > > - {title} - the same as description - but another file for title
> > > > > > > - {Status} - rnadom staus (overall there are 3 statuses
> > > > > > > - {First\_RndNumber} - random nuber between 1 to 800
> > > > > > > - The all other parameters contain random number
> > > > > > > 
> > > > > > > _Search cases:_  
> > > > > > > you can see in the attached file(SearchCases.txt) the search  
> > > > > > > operation, please let me know if you need clarification for this action.
> > > > > > > 
> > > > > > > _Workload:_  
> > > > > > > Create cases was executed with 20 concurrent users which create case  
> > > > > > > every ~ 4 seconds  
> > > > > > > Search cases was executed with 10 concurrent users which executed the  
> > > > > > > search every ~25 seconds.  
> > > > > > > The create and the search run on the same time on the system.
> > > > > > > 
> > > > > > > Thanks,  
> > > > > > > Shay
> > > > > > > 
> > > > > > > On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon \<  
> > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > 
> > > > > > > > This certainly does not look good. I have just run a test against  
> > > > > > > > 0.8 with 2k TPS index and search operations (simple ones) and the open files  
> > > > > > > > system seems to be good (constant around 80) (with an FS gateway).
> > > > > > > > 
> > > > > > > > Maybe you uncovered a problem here that I can't recreate. Can you  
> > > > > > > > explain more about the test that you run? Go into details regarding the  
> > > > > > > > operations that you do. Which search requests, what parameters? Do you use  
> > > > > > > > scroll search request?
> > > > > > > > 
> > > > > > > > Maybe you can create a simple test case that I can run here?
> > > > > > > > 
> > > > > > > > -shay.banon
> > > > > > > > 
> > > > > > > > On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > > > 
> > > > > > > > > Hi,  
> > > > > > > > > I executed a new perfromance test for the Elasticsearch, and have  
> > > > > > > > > the same results.  
> > > > > > > > > Attached please find 2 files:
> > > > > > > > > 
> > > > > > > > > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend of  
> > > > > > > > > the number of open files (_lsof | wc -l_ is running on the system  
> > > > > > > > > every 1 minute).  
> > > > > > > > > btw  
> > > > > > > > > Please look in the following output:  
> > > > > > > > > [root]# \*lsof | wc -l  
> > > > > > > > > \*23917  
> > > > > > > > > [root]# \*lsof | grep elastic | wc -l  
> > > > > > > > > \*23376  
> > > > > > > > > 2.The second file (ElasticSearchOpenFiles.zip) contain the output  
> > > > > > > > > from the following command: lsof | grep elastic
> > > > > > > > > 
> > > > > > > > > Thanks,  
> > > > > > > > > Shay
> > > > > > > > > 
> > > > > > > > > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > This should not happen. Is it still running? Can you mail me the a  
> > > > > > > > > > sample of the lsof output?
> > > > > > > > > > 
> > > > > > > > > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi,  
> > > > > > > > > > > I'm conducting a test for create and search indexes(5 case  
> > > > > > > > > > > creation  
> > > > > > > > > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > > > > > > > > I monitored the number of open files every 1 minute (lsof | wc  
> > > > > > > > > > > -l) and  
> > > > > > > > > > > it seems there is a gradual growth in the number of open files  
> > > > > > > > > > > though  
> > > > > > > > > > > there is no increasing in the load over the time.  
> > > > > > > > > > > After ~ 20 hours 17K open files were on the system, 15K of them  
> > > > > > > > > > > belong  
> > > > > > > > > > > to the Elasticsearch.  
> > > > > > > > > > > Is it a noraml behaior of the Elasticsearch?  
> > > > > > > > > > > I have a file with the list of the open files and excel which  
> > > > > > > > > > > shows  
> > > > > > > > > > > the trend of the number of open files on the system, so if you  
> > > > > > > > > > > would  
> > > > > > > > > > > like to see it, please send me a mail and i'll send you these  
> > > > > > > > > > > files.  
> > > > > > > > > > > Thanks,  
> > > > > > > > > > > Shay

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 25, 2010, 8:04pm UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/13 "2010-06-25T20:04:45Z")

</div>

Hi mate,

Did you get a chance to have a look at this? I tried to recreate this,  
but I can't, so would love to see if you can create something that I can  
recreate (against either 0.8 or better, master) since if this is a real  
problem, I would love to tackle it.

thanks,  
shay.banon

On Sun, Jun 6, 2010 at 1:37 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Not really, this is usually very low level problem, I need to be able to  
> recreate it and basically debug it...
> 
> On Sun, Jun 6, 2010 at 12:19 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> 
> > I can try do this but it'll probably take some time.  
> > Maybe you have any idea regarding the log level- maybe i can run the same  
> > test with a different log level which will give me more indecation about the  
> > cause for many open files?
> > 
> > On Fri, Jun 4, 2010 at 9:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> > 
> > > Is there a chance that you can create a simple test case that I can run  
> > > that simulates this, it would really help to speed things up. Something that  
> > > even uses bash script with curls, or simple Java program, or something  
> > > similar?
> > > 
> > > On Thu, Jun 3, 2010 at 2:15 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > 
> > > > I don't think i made any special configuration fore creating the case  
> > > > , it looks like:  
> > > > http://{ServerName}:{Port}/_{UserId}_/CASE/{UserId}{objectId}{UserIdhttp://{servername}%3A{port}/{UserId}/CASE/{UserId}{objectId}{UserId  
> > > > }  
> > > > when {UserId} is a constant number for each user.  
> > > > I executed the optimize after performance test without it which shows  
> > > > the same phenomenon.
> > > > 
> > > > On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon \<  
> > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > 
> > > > > One more thing, any special configuration when creating an index? Also,  
> > > > > the big difference, I think from what I did to what you did is running  
> > > > > optimize, can you try without it and see if it still happens? You don't  
> > > > > really need to call either.
> > > > > 
> > > > > On Thu, Jun 3, 2010 at 1:53 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > > 
> > > > > > Attached is the yml file.  
> > > > > > I'm running on a single node.  
> > > > > > I have 20 users for each one of them i created an index (20 indexes in  
> > > > > > the total)  
> > > > > > btw,  
> > > > > > optimize+flush were executed every 15 minutes  
> > > > > > Thanks,  
> > > > > > Shay
> > > > > > 
> > > > > > On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon \<  
> > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > 
> > > > > > > ok. I will try and run something like this, but, if you want to move  
> > > > > > > things faster, it would help if you can create a reproduction that I can  
> > > > > > > run. Also, how many nodes are running? Can you share you configuration? Do  
> > > > > > > you create an index per user?
> > > > > > > 
> > > > > > > -shay.banon
> > > > > > > 
> > > > > > > On Thu, Jun 3, 2010 at 9:37 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > > 
> > > > > > > > The test was executed with load runner.  
> > > > > > > > I'll try to describe it.  
> > > > > > > > You can see in the attached file(create\_case.txt) the http call for  
> > > > > > > > _create cases_.  
> > > > > > > > Parameters description:
> > > > > > > > 
> > > > > > > > - {UserId} - we have 20 concurrent users each of them have a  
> > > > > > > > unique {UserId}
> > > > > > > > - {objectId} - number which increasing every iteration in 1
> > > > > > > > - {Description1}- we have a file with ~ 70K lines so in  
> > > > > > > > each iteration the Load Runner choose 1 random line for description
> > > > > > > > - {title} - the same as description - but another file for title
> > > > > > > > - {Status} - rnadom staus (overall there are 3 statuses
> > > > > > > > - {First\_RndNumber} - random nuber between 1 to 800
> > > > > > > > - The all other parameters contain random number
> > > > > > > > 
> > > > > > > > _Search cases:_  
> > > > > > > > you can see in the attached file(SearchCases.txt) the search  
> > > > > > > > operation, please let me know if you need clarification for this action.
> > > > > > > > 
> > > > > > > > _Workload:_  
> > > > > > > > Create cases was executed with 20 concurrent users which create case  
> > > > > > > > every ~ 4 seconds  
> > > > > > > > Search cases was executed with 10 concurrent users which executed  
> > > > > > > > the search every ~25 seconds.  
> > > > > > > > The create and the search run on the same time on the system.
> > > > > > > > 
> > > > > > > > Thanks,  
> > > > > > > > Shay
> > > > > > > > 
> > > > > > > > On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon \<  
> > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > 
> > > > > > > > > This certainly does not look good. I have just run a test against  
> > > > > > > > > 0.8 with 2k TPS index and search operations (simple ones) and the open files  
> > > > > > > > > system seems to be good (constant around 80) (with an FS gateway).
> > > > > > > > > 
> > > > > > > > > Maybe you uncovered a problem here that I can't recreate. Can you  
> > > > > > > > > explain more about the test that you run? Go into details regarding the  
> > > > > > > > > operations that you do. Which search requests, what parameters? Do you use  
> > > > > > > > > scroll search request?
> > > > > > > > > 
> > > > > > > > > Maybe you can create a simple test case that I can run here?
> > > > > > > > > 
> > > > > > > > > -shay.banon
> > > > > > > > > 
> > > > > > > > > On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > > > > 
> > > > > > > > > > Hi,  
> > > > > > > > > > I executed a new perfromance test for the Elasticsearch, and have  
> > > > > > > > > > the same results.  
> > > > > > > > > > Attached please find 2 files:
> > > > > > > > > > 
> > > > > > > > > > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend  
> > > > > > > > > > of the number of open files (_lsof | wc -l_ is running on the  
> > > > > > > > > > system every 1 minute).  
> > > > > > > > > > btw  
> > > > > > > > > > Please look in the following output:  
> > > > > > > > > > [root]# \*lsof | wc -l  
> > > > > > > > > > \*23917  
> > > > > > > > > > [root]# \*lsof | grep elastic | wc -l  
> > > > > > > > > > \*23376  
> > > > > > > > > > 2.The second file (ElasticSearchOpenFiles.zip) contain the output  
> > > > > > > > > > from the following command: lsof | grep elastic
> > > > > > > > > > 
> > > > > > > > > > Thanks,  
> > > > > > > > > > Shay
> > > > > > > > > > 
> > > > > > > > > > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > > > 
> > > > > > > > > > > This should not happen. Is it still running? Can you mail me the  
> > > > > > > > > > > a sample of the lsof output?
> > > > > > > > > > > 
> > > > > > > > > > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hi,  
> > > > > > > > > > > > I'm conducting a test for create and search indexes(5 case  
> > > > > > > > > > > > creation  
> > > > > > > > > > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > > > > > > > > > I monitored the number of open files every 1 minute (lsof | wc  
> > > > > > > > > > > > -l) and  
> > > > > > > > > > > > it seems there is a gradual growth in the number of open files  
> > > > > > > > > > > > though  
> > > > > > > > > > > > there is no increasing in the load over the time.  
> > > > > > > > > > > > After ~ 20 hours 17K open files were on the system, 15K of them  
> > > > > > > > > > > > belong  
> > > > > > > > > > > > to the Elasticsearch.  
> > > > > > > > > > > > Is it a noraml behaior of the Elasticsearch?  
> > > > > > > > > > > > I have a file with the list of the open files and excel which  
> > > > > > > > > > > > shows  
> > > > > > > > > > > > the trend of the number of open files on the system, so if you  
> > > > > > > > > > > > would  
> > > > > > > > > > > > like to see it, please send me a mail and i'll send you these  
> > > > > > > > > > > > files.  
> > > > > > > > > > > > Thanks,  
> > > > > > > > > > > > Shay

---

<div class="post-metadata">

**Author:** ![shay\_katz](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shay\_katz](https://discuss.elastic.co/u/shay_katz)\
**Post date:** [June 26, 2010, 6:19pm UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/14 "2010-06-26T18:19:05Z")

</div>

Hi,  
I hope that during next make i'll conduct similar test via java API, i'll  
monitor the heap size and the number of open files and will let you know the  
results.  
Thanks,  
Shay

On Fri, Jun 25, 2010 at 11:04 PM, Shay Banon  
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Hi mate,
> 
> Did you get a chance to have a look at this? I tried to recreate this,  
> but I can't, so would love to see if you can create something that I can  
> recreate (against either 0.8 or better, master) since if this is a real  
> problem, I would love to tackle it.
> 
> thanks,  
> shay.banon
> 
> On Sun, Jun 6, 2010 at 1:37 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > Not really, this is usually very low level problem, I need to be able to  
> > recreate it and basically debug it...
> > 
> > On Sun, Jun 6, 2010 at 12:19 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > 
> > > I can try do this but it'll probably take some time.  
> > > Maybe you have any idea regarding the log level- maybe i can run the same  
> > > test with a different log level which will give me more indecation about the  
> > > cause for many open files?
> > > 
> > > On Fri, Jun 4, 2010 at 9:17 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > 
> > > > wrote:
> > > 
> > > > Is there a chance that you can create a simple test case that I can run  
> > > > that simulates this, it would really help to speed things up. Something that  
> > > > even uses bash script with curls, or simple Java program, or something  
> > > > similar?
> > > > 
> > > > On Thu, Jun 3, 2010 at 2:15 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com) wrote:
> > > > 
> > > > > I don't think i made any special configuration fore creating the case  
> > > > > , it looks like:  
> > > > > http://{ServerName}:{Port}/_{UserId}_/CASE/{UserId}{objectId}{UserIdhttp://{servername}%3A{port}/{UserId}/CASE/{UserId}{objectId}{UserId  
> > > > > }  
> > > > > when {UserId} is a constant number for each user.  
> > > > > I executed the optimize after performance test without it which shows  
> > > > > the same phenomenon.
> > > > > 
> > > > > On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon \<  
> > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > 
> > > > > > One more thing, any special configuration when creating an index?  
> > > > > > Also, the big difference, I think from what I did to what you did is running  
> > > > > > optimize, can you try without it and see if it still happens? You don't  
> > > > > > really need to call either.
> > > > > > 
> > > > > > On Thu, Jun 3, 2010 at 1:53 PM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > 
> > > > > > > Attached is the yml file.  
> > > > > > > I'm running on a single node.  
> > > > > > > I have 20 users for each one of them i created an index (20 indexes  
> > > > > > > in the total)  
> > > > > > > btw,  
> > > > > > > optimize+flush were executed every 15 minutes  
> > > > > > > Thanks,  
> > > > > > > Shay
> > > > > > > 
> > > > > > > On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon \<  
> > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > 
> > > > > > > > ok. I will try and run something like this, but, if you want to move  
> > > > > > > > things faster, it would help if you can create a reproduction that I can  
> > > > > > > > run. Also, how many nodes are running? Can you share you configuration? Do  
> > > > > > > > you create an index per user?
> > > > > > > > 
> > > > > > > > -shay.banon
> > > > > > > > 
> > > > > > > > On Thu, Jun 3, 2010 at 9:37 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > > > 
> > > > > > > > > The test was executed with load runner.  
> > > > > > > > > I'll try to describe it.  
> > > > > > > > > You can see in the attached file(create\_case.txt) the http call for  
> > > > > > > > > _create cases_.  
> > > > > > > > > Parameters description:
> > > > > > > > > 
> > > > > > > > > - {UserId} - we have 20 concurrent users each of them have a  
> > > > > > > > > unique {UserId}
> > > > > > > > > - {objectId} - number which increasing every iteration in 1
> > > > > > > > > - {Description1}- we have a file with ~ 70K lines so in  
> > > > > > > > > each iteration the Load Runner choose 1 random line for description
> > > > > > > > > - {title} - the same as description - but another file for  
> > > > > > > > > title
> > > > > > > > > - {Status} - rnadom staus (overall there are 3 statuses
> > > > > > > > > - {First\_RndNumber} - random nuber between 1 to 800
> > > > > > > > > - The all other parameters contain random number
> > > > > > > > > 
> > > > > > > > > _Search cases:_  
> > > > > > > > > you can see in the attached file(SearchCases.txt) the search  
> > > > > > > > > operation, please let me know if you need clarification for this action.
> > > > > > > > > 
> > > > > > > > > _Workload:_  
> > > > > > > > > Create cases was executed with 20 concurrent users which create  
> > > > > > > > > case every ~ 4 seconds  
> > > > > > > > > Search cases was executed with 10 concurrent users which executed  
> > > > > > > > > the search every ~25 seconds.  
> > > > > > > > > The create and the search run on the same time on the system.
> > > > > > > > > 
> > > > > > > > > Thanks,  
> > > > > > > > > Shay
> > > > > > > > > 
> > > > > > > > > On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon \<  
> > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > This certainly does not look good. I have just run a test against  
> > > > > > > > > > 0.8 with 2k TPS index and search operations (simple ones) and the open files  
> > > > > > > > > > system seems to be good (constant around 80) (with an FS gateway).
> > > > > > > > > > 
> > > > > > > > > > Maybe you uncovered a problem here that I can't recreate. Can you  
> > > > > > > > > > explain more about the test that you run? Go into details regarding the  
> > > > > > > > > > operations that you do. Which search requests, what parameters? Do you use  
> > > > > > > > > > scroll search request?
> > > > > > > > > > 
> > > > > > > > > > Maybe you can create a simple test case that I can run here?
> > > > > > > > > > 
> > > > > > > > > > -shay.banon
> > > > > > > > > > 
> > > > > > > > > > On Wed, Jun 2, 2010 at 11:16 AM, shay katz [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi,  
> > > > > > > > > > > I executed a new perfromance test for the Elasticsearch, and  
> > > > > > > > > > > have the same results.  
> > > > > > > > > > > Attached please find 2 files:
> > > > > > > > > > > 
> > > > > > > > > > > 1. Elastic\_Search\_NUMBER\_OF\_OPEN\_FILES.xlsx - Describe the trend  
> > > > > > > > > > > of the number of open files (_lsof | wc -l_ is running on the  
> > > > > > > > > > > system every 1 minute).  
> > > > > > > > > > > btw  
> > > > > > > > > > > Please look in the following output:  
> > > > > > > > > > > [root]# \*lsof | wc -l  
> > > > > > > > > > > \*23917  
> > > > > > > > > > > [root]# \*lsof | grep elastic | wc -l  
> > > > > > > > > > > \*23376  
> > > > > > > > > > > 2.The second file (ElasticSearchOpenFiles.zip) contain the output  
> > > > > > > > > > > from the following command: lsof | grep elastic
> > > > > > > > > > > 
> > > > > > > > > > > Thanks,  
> > > > > > > > > > > Shay
> > > > > > > > > > > 
> > > > > > > > > > > On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > This should not happen. Is it still running? Can you mail me the  
> > > > > > > > > > > > a sample of the lsof output?
> > > > > > > > > > > > 
> > > > > > > > > > > > On Tue, Jun 1, 2010 at 2:23 PM, Shay [katzshay1@gmail.com](mailto:katzshay1@gmail.com)wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi,  
> > > > > > > > > > > > > I'm conducting a test for create and search indexes(5 case  
> > > > > > > > > > > > > creation  
> > > > > > > > > > > > > every 1 seconds and 1 search every ~ 2 seconds.  
> > > > > > > > > > > > > I monitored the number of open files every 1 minute (lsof | wc  
> > > > > > > > > > > > > -l) and  
> > > > > > > > > > > > > it seems there is a gradual growth in the number of open files  
> > > > > > > > > > > > > though  
> > > > > > > > > > > > > there is no increasing in the load over the time.  
> > > > > > > > > > > > > After ~ 20 hours 17K open files were on the system, 15K of them  
> > > > > > > > > > > > > belong  
> > > > > > > > > > > > > to the Elasticsearch.  
> > > > > > > > > > > > > Is it a noraml behaior of the Elasticsearch?  
> > > > > > > > > > > > > I have a file with the list of the open files and excel which  
> > > > > > > > > > > > > shows  
> > > > > > > > > > > > > the trend of the number of open files on the system, so if you  
> > > > > > > > > > > > > would  
> > > > > > > > > > > > > like to see it, please send me a mail and i'll send you these  
> > > > > > > > > > > > > files.  
> > > > > > > > > > > > > Thanks,  
> > > > > > > > > > > > > Shay

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:22am UTC](https://discuss.elastic.co/t/gradual-growth-in-the-number-of-open-files-in-linux/2985/15 "2017-07-06T04:22:58Z")

</div>