Gradual Growth in the number of open files in linux

Elastic Stack Elasticsearch
1

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l) and
it seems there is a gradual growth in the number of open files though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them belong
to the elastic search.
Is it a noraml behaior of the elastic search?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you would
like to see it, please send me a mail and i'll send you these files.
Thanks,
Shay

2

This should not happen. Is it still running? Can you mail me the a sample of
the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l) and
it seems there is a gradual growth in the number of open files though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you would
like to see it, please send me a mail and i'll send you these files.
Thanks,
Shay

3

Hi,
I executed a new perfromance test for the Elasticsearch, and have the same
results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of the
    number of open files (lsof | wc -l is running on the system every 1
    minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output from the
    following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon shay.banon@elasticsearch.comwrote:

This should not happen. Is it still running? Can you mail me the a sample
of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l) and
it seems there is a gradual growth in the number of open files though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you would
like to see it, please send me a mail and i'll send you these files.
Thanks,
Shay

4

This certainly does not look good. I have just run a test against 0.8 with
2k TPS index and search operations (simple ones) and the open files system
seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you explain
more about the test that you run? Go into details regarding the operations
that you do. Which search requests, what parameters? Do you use scroll
search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.com wrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have the same
results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of the
    number of open files (lsof | wc -l is running on the system every 1
    minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output from the
    following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon shay.banon@elasticsearch.comwrote:

This should not happen. Is it still running? Can you mail me the a sample
of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l) and
it seems there is a gradual growth in the number of open files though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you would
like to see it, please send me a mail and i'll send you these files.
Thanks,
Shay

5

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for create
cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a unique
    {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in each iteration
    the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search operation,
please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create case every ~
4 seconds
Search cases was executed with 10 concurrent users which executed the search
every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon shay.banon@elasticsearch.comwrote:

This certainly does not look good. I have just run a test against 0.8 with
2k TPS index and search operations (simple ones) and the open files system
seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you explain
more about the test that you run? Go into details regarding the operations
that you do. Which search requests, what parameters? Do you use scroll
search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.com wrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have the
same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of the
    number of open files (lsof | wc -l is running on the system every 1
    minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output from the
    following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the a sample
of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l) and
it seems there is a gradual growth in the number of open files though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you would
like to see it, please send me a mail and i'll send you these files.
Thanks,
Shay

6

ok. I will try and run something like this, but, if you want to move things
faster, it would help if you can create a reproduction that I can run. Also,
how many nodes are running? Can you share you configuration? Do you create
an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz katzshay1@gmail.com wrote:

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for create
cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a unique
    {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in each iteration
    the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search operation,
please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create case every
~ 4 seconds
Search cases was executed with 10 concurrent users which executed the
search every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon shay.banon@elasticsearch.comwrote:

This certainly does not look good. I have just run a test against 0.8 with
2k TPS index and search operations (simple ones) and the open files system
seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you explain
more about the test that you run? Go into details regarding the operations
that you do. Which search requests, what parameters? Do you use scroll
search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.com wrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have the
same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of the
    number of open files (lsof | wc -l is running on the system every 1
    minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output from
    the following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the a
sample of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l) and
it seems there is a gradual growth in the number of open files though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you would
like to see it, please send me a mail and i'll send you these files.
Thanks,
Shay

7

Attached is the yml file.
I'm running on a single node.
I have 20 users for each one of them i created an index (20 indexes in the
total)
btw,
optimize+flush were executed every 15 minutes
Thanks,
Shay

On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon shay.banon@elasticsearch.comwrote:

ok. I will try and run something like this, but, if you want to move things
faster, it would help if you can create a reproduction that I can run. Also,
how many nodes are running? Can you share you configuration? Do you create
an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz katzshay1@gmail.com wrote:

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for create
cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a unique
    {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in each iteration
    the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search operation,
please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create case every
~ 4 seconds
Search cases was executed with 10 concurrent users which executed the
search every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon <shay.banon@elasticsearch.com

wrote:

This certainly does not look good. I have just run a test against 0.8
with 2k TPS index and search operations (simple ones) and the open files
system seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you explain
more about the test that you run? Go into details regarding the operations
that you do. Which search requests, what parameters? Do you use scroll
search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.com wrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have the
same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of the
    number of open files (lsof | wc -l is running on the system every 1
    minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output from
    the following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the a
sample of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l) and
it seems there is a gradual growth in the number of open files though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you would
like to see it, please send me a mail and i'll send you these files.
Thanks,
Shay

8

One more thing, any special configuration when creating an index? Also, the
big difference, I think from what I did to what you did is running optimize,
can you try without it and see if it still happens? You don't really need to
call either.

On Thu, Jun 3, 2010 at 1:53 PM, shay katz katzshay1@gmail.com wrote:

Attached is the yml file.
I'm running on a single node.
I have 20 users for each one of them i created an index (20 indexes in the
total)
btw,
optimize+flush were executed every 15 minutes
Thanks,
Shay

On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon shay.banon@elasticsearch.comwrote:

ok. I will try and run something like this, but, if you want to move
things faster, it would help if you can create a reproduction that I can
run. Also, how many nodes are running? Can you share you configuration? Do
you create an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz katzshay1@gmail.com wrote:

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for create
cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a unique
    {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in
    each iteration the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search operation,
please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create case
every ~ 4 seconds
Search cases was executed with 10 concurrent users which executed the
search every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This certainly does not look good. I have just run a test against 0.8
with 2k TPS index and search operations (simple ones) and the open files
system seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you
explain more about the test that you run? Go into details regarding the
operations that you do. Which search requests, what parameters? Do you use
scroll search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.com wrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have the
same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of the
    number of open files (lsof | wc -l is running on the system every 1
    minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output from
    the following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the a
sample of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l)
and
it seems there is a gradual growth in the number of open files though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them
belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you would
like to see it, please send me a mail and i'll send you these files.
Thanks,
Shay

9

I don't think i made any special configuration fore creating the case , it
looks like:
http://{ServerName}:{Port}/{UserId}/CASE/{UserId}{objectId}{UserIdhttp://{ServerName}:{Port}/{UserId}/CASE/{UserId}{objectId}{UserId
}
when {UserId} is a constant number for each user.
I executed the optimize after performance test without it which shows the
same phenomenon.

On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon shay.banon@elasticsearch.comwrote:

One more thing, any special configuration when creating an index? Also, the
big difference, I think from what I did to what you did is running optimize,
can you try without it and see if it still happens? You don't really need to
call either.

On Thu, Jun 3, 2010 at 1:53 PM, shay katz katzshay1@gmail.com wrote:

Attached is the yml file.
I'm running on a single node.
I have 20 users for each one of them i created an index (20 indexes in the
total)
btw,
optimize+flush were executed every 15 minutes
Thanks,
Shay

On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

ok. I will try and run something like this, but, if you want to move
things faster, it would help if you can create a reproduction that I can
run. Also, how many nodes are running? Can you share you configuration? Do
you create an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz katzshay1@gmail.com wrote:

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for create
cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a unique
    {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in
    each iteration the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search operation,
please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create case
every ~ 4 seconds
Search cases was executed with 10 concurrent users which executed the
search every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This certainly does not look good. I have just run a test against 0.8
with 2k TPS index and search operations (simple ones) and the open files
system seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you
explain more about the test that you run? Go into details regarding the
operations that you do. Which search requests, what parameters? Do you use
scroll search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.comwrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have the
same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of
    the number of open files (lsof | wc -l is running on the system
    every 1 minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output from
    the following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the a
sample of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l)
and
it seems there is a gradual growth in the number of open files
though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them
belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you would
like to see it, please send me a mail and i'll send you these files.
Thanks,
Shay

10

Is there a chance that you can create a simple test case that I can run that
simulates this, it would really help to speed things up. Something that even
uses bash script with curls, or simple Java program, or something similar?

On Thu, Jun 3, 2010 at 2:15 PM, shay katz katzshay1@gmail.com wrote:

I don't think i made any special configuration fore creating the case , it
looks like:
http://{ServerName}:{Port}/{UserId}/CASE/{UserId}{objectId}{UserIdhttp://{ServerName}%3A{Port}/{UserId}/CASE/{UserId}{objectId}{UserId
}
when {UserId} is a constant number for each user.
I executed the optimize after performance test without it which shows the
same phenomenon.

On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon shay.banon@elasticsearch.comwrote:

One more thing, any special configuration when creating an index? Also,
the big difference, I think from what I did to what you did is running
optimize, can you try without it and see if it still happens? You don't
really need to call either.

On Thu, Jun 3, 2010 at 1:53 PM, shay katz katzshay1@gmail.com wrote:

Attached is the yml file.
I'm running on a single node.
I have 20 users for each one of them i created an index (20 indexes in
the total)
btw,
optimize+flush were executed every 15 minutes
Thanks,
Shay

On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

ok. I will try and run something like this, but, if you want to move
things faster, it would help if you can create a reproduction that I can
run. Also, how many nodes are running? Can you share you configuration? Do
you create an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz katzshay1@gmail.com wrote:

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for create
cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a unique
    {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in
    each iteration the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search operation,
please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create case
every ~ 4 seconds
Search cases was executed with 10 concurrent users which executed the
search every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This certainly does not look good. I have just run a test against 0.8
with 2k TPS index and search operations (simple ones) and the open files
system seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you
explain more about the test that you run? Go into details regarding the
operations that you do. Which search requests, what parameters? Do you use
scroll search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.comwrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have
the same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of
    the number of open files (lsof | wc -l is running on the system
    every 1 minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output
    from the following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the a
sample of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l)
and
it seems there is a gradual growth in the number of open files
though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them
belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which shows
the trend of the number of open files on the system, so if you
would
like to see it, please send me a mail and i'll send you these
files.
Thanks,
Shay

11

I can try do this but it'll probably take some time.
Maybe you have any idea regarding the log level- maybe i can run the same
test with a different log level which will give me more indecation about the
cause for many open files?

On Fri, Jun 4, 2010 at 9:17 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Is there a chance that you can create a simple test case that I can run
that simulates this, it would really help to speed things up. Something that
even uses bash script with curls, or simple Java program, or something
similar?

On Thu, Jun 3, 2010 at 2:15 PM, shay katz katzshay1@gmail.com wrote:

I don't think i made any special configuration fore creating the case ,
it looks like:
http://{ServerName}:{Port}/{UserId}/CASE/{UserId}{objectId}{UserIdhttp://{servername}%3A{port}/{UserId}/CASE/{UserId}{objectId}{UserId
}
when {UserId} is a constant number for each user.
I executed the optimize after performance test without it which shows the
same phenomenon.

On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon shay.banon@elasticsearch.comwrote:

One more thing, any special configuration when creating an index? Also,
the big difference, I think from what I did to what you did is running
optimize, can you try without it and see if it still happens? You don't
really need to call either.

On Thu, Jun 3, 2010 at 1:53 PM, shay katz katzshay1@gmail.com wrote:

Attached is the yml file.
I'm running on a single node.
I have 20 users for each one of them i created an index (20 indexes in
the total)
btw,
optimize+flush were executed every 15 minutes
Thanks,
Shay

On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

ok. I will try and run something like this, but, if you want to move
things faster, it would help if you can create a reproduction that I can
run. Also, how many nodes are running? Can you share you configuration? Do
you create an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz katzshay1@gmail.com wrote:

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for create
cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a
    unique {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in
    each iteration the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search
operation, please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create case
every ~ 4 seconds
Search cases was executed with 10 concurrent users which executed the
search every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This certainly does not look good. I have just run a test against 0.8
with 2k TPS index and search operations (simple ones) and the open files
system seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you
explain more about the test that you run? Go into details regarding the
operations that you do. Which search requests, what parameters? Do you use
scroll search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.comwrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have
the same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of
    the number of open files (lsof | wc -l is running on the system
    every 1 minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output
    from the following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the a
sample of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case
creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc -l)
and
it seems there is a gradual growth in the number of open files
though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them
belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which
shows
the trend of the number of open files on the system, so if you
would
like to see it, please send me a mail and i'll send you these
files.
Thanks,
Shay

12

Not really, this is usually very low level problem, I need to be able to
recreate it and basically debug it...

On Sun, Jun 6, 2010 at 12:19 AM, shay katz katzshay1@gmail.com wrote:

I can try do this but it'll probably take some time.
Maybe you have any idea regarding the log level- maybe i can run the same
test with a different log level which will give me more indecation about the
cause for many open files?

On Fri, Jun 4, 2010 at 9:17 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Is there a chance that you can create a simple test case that I can run
that simulates this, it would really help to speed things up. Something that
even uses bash script with curls, or simple Java program, or something
similar?

On Thu, Jun 3, 2010 at 2:15 PM, shay katz katzshay1@gmail.com wrote:

I don't think i made any special configuration fore creating the case ,
it looks like:
http://{ServerName}:{Port}/{UserId}/CASE/{UserId}{objectId}{UserIdhttp://{servername}%3A{port}/{UserId}/CASE/{UserId}{objectId}{UserId
}
when {UserId} is a constant number for each user.
I executed the optimize after performance test without it which shows the
same phenomenon.

On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon <shay.banon@elasticsearch.com

wrote:

One more thing, any special configuration when creating an index? Also,
the big difference, I think from what I did to what you did is running
optimize, can you try without it and see if it still happens? You don't
really need to call either.

On Thu, Jun 3, 2010 at 1:53 PM, shay katz katzshay1@gmail.com wrote:

Attached is the yml file.
I'm running on a single node.
I have 20 users for each one of them i created an index (20 indexes in
the total)
btw,
optimize+flush were executed every 15 minutes
Thanks,
Shay

On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

ok. I will try and run something like this, but, if you want to move
things faster, it would help if you can create a reproduction that I can
run. Also, how many nodes are running? Can you share you configuration? Do
you create an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz katzshay1@gmail.comwrote:

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for
create cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a
    unique {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in
    each iteration the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search
operation, please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create case
every ~ 4 seconds
Search cases was executed with 10 concurrent users which executed the
search every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This certainly does not look good. I have just run a test against
0.8 with 2k TPS index and search operations (simple ones) and the open files
system seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you
explain more about the test that you run? Go into details regarding the
operations that you do. Which search requests, what parameters? Do you use
scroll search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.comwrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have
the same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend of
    the number of open files (lsof | wc -l is running on the system
    every 1 minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output
    from the following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the a
sample of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.com wrote:

Hi,
I'm conducting a test for create and search indexes(5 case
creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc
-l) and
it seems there is a gradual growth in the number of open files
though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them
belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which
shows
the trend of the number of open files on the system, so if you
would
like to see it, please send me a mail and i'll send you these
files.
Thanks,
Shay

13

Hi mate,

Did you get a chance to have a look at this? I tried to recreate this,
but I can't, so would love to see if you can create something that I can
recreate (against either 0.8 or better, master) since if this is a real
problem, I would love to tackle it.

thanks,
shay.banon

On Sun, Jun 6, 2010 at 1:37 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Not really, this is usually very low level problem, I need to be able to
recreate it and basically debug it...

On Sun, Jun 6, 2010 at 12:19 AM, shay katz katzshay1@gmail.com wrote:

I can try do this but it'll probably take some time.
Maybe you have any idea regarding the log level- maybe i can run the same
test with a different log level which will give me more indecation about the
cause for many open files?

On Fri, Jun 4, 2010 at 9:17 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Is there a chance that you can create a simple test case that I can run
that simulates this, it would really help to speed things up. Something that
even uses bash script with curls, or simple Java program, or something
similar?

On Thu, Jun 3, 2010 at 2:15 PM, shay katz katzshay1@gmail.com wrote:

I don't think i made any special configuration fore creating the case
, it looks like:
http://{ServerName}:{Port}/{UserId}/CASE/{UserId}{objectId}{UserIdhttp://{servername}%3A{port}/{UserId}/CASE/{UserId}{objectId}{UserId
}
when {UserId} is a constant number for each user.
I executed the optimize after performance test without it which shows
the same phenomenon.

On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

One more thing, any special configuration when creating an index? Also,
the big difference, I think from what I did to what you did is running
optimize, can you try without it and see if it still happens? You don't
really need to call either.

On Thu, Jun 3, 2010 at 1:53 PM, shay katz katzshay1@gmail.com wrote:

Attached is the yml file.
I'm running on a single node.
I have 20 users for each one of them i created an index (20 indexes in
the total)
btw,
optimize+flush were executed every 15 minutes
Thanks,
Shay

On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

ok. I will try and run something like this, but, if you want to move
things faster, it would help if you can create a reproduction that I can
run. Also, how many nodes are running? Can you share you configuration? Do
you create an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz katzshay1@gmail.comwrote:

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for
create cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a
    unique {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in
    each iteration the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search
operation, please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create case
every ~ 4 seconds
Search cases was executed with 10 concurrent users which executed
the search every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This certainly does not look good. I have just run a test against
0.8 with 2k TPS index and search operations (simple ones) and the open files
system seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you
explain more about the test that you run? Go into details regarding the
operations that you do. Which search requests, what parameters? Do you use
scroll search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.comwrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and have
the same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend
    of the number of open files (lsof | wc -l is running on the
    system every 1 minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output
    from the following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the
a sample of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.comwrote:

Hi,
I'm conducting a test for create and search indexes(5 case
creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc
-l) and
it seems there is a gradual growth in the number of open files
though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them
belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which
shows
the trend of the number of open files on the system, so if you
would
like to see it, please send me a mail and i'll send you these
files.
Thanks,
Shay

14

Hi,
I hope that during next make i'll conduct similar test via java API, i'll
monitor the heap size and the number of open files and will let you know the
results.
Thanks,
Shay

On Fri, Jun 25, 2010 at 11:04 PM, Shay Banon
shay.banon@elasticsearch.comwrote:

Hi mate,

Did you get a chance to have a look at this? I tried to recreate this,
but I can't, so would love to see if you can create something that I can
recreate (against either 0.8 or better, master) since if this is a real
problem, I would love to tackle it.

thanks,
shay.banon

On Sun, Jun 6, 2010 at 1:37 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Not really, this is usually very low level problem, I need to be able to
recreate it and basically debug it...

On Sun, Jun 6, 2010 at 12:19 AM, shay katz katzshay1@gmail.com wrote:

I can try do this but it'll probably take some time.
Maybe you have any idea regarding the log level- maybe i can run the same
test with a different log level which will give me more indecation about the
cause for many open files?

On Fri, Jun 4, 2010 at 9:17 PM, Shay Banon <shay.banon@elasticsearch.com

wrote:

Is there a chance that you can create a simple test case that I can run
that simulates this, it would really help to speed things up. Something that
even uses bash script with curls, or simple Java program, or something
similar?

On Thu, Jun 3, 2010 at 2:15 PM, shay katz katzshay1@gmail.com wrote:

I don't think i made any special configuration fore creating the case
, it looks like:
http://{ServerName}:{Port}/{UserId}/CASE/{UserId}{objectId}{UserIdhttp://{servername}%3A{port}/{UserId}/CASE/{UserId}{objectId}{UserId
}
when {UserId} is a constant number for each user.
I executed the optimize after performance test without it which shows
the same phenomenon.

On Thu, Jun 3, 2010 at 2:04 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

One more thing, any special configuration when creating an index?
Also, the big difference, I think from what I did to what you did is running
optimize, can you try without it and see if it still happens? You don't
really need to call either.

On Thu, Jun 3, 2010 at 1:53 PM, shay katz katzshay1@gmail.comwrote:

Attached is the yml file.
I'm running on a single node.
I have 20 users for each one of them i created an index (20 indexes
in the total)
btw,
optimize+flush were executed every 15 minutes
Thanks,
Shay

On Thu, Jun 3, 2010 at 12:44 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

ok. I will try and run something like this, but, if you want to move
things faster, it would help if you can create a reproduction that I can
run. Also, how many nodes are running? Can you share you configuration? Do
you create an index per user?

-shay.banon

On Thu, Jun 3, 2010 at 9:37 AM, shay katz katzshay1@gmail.comwrote:

The test was executed with load runner.
I'll try to describe it.
You can see in the attached file(create_case.txt) the http call for
create cases.
Parameters description:

  • {UserId} - we have 20 concurrent users each of them have a
    unique {UserId}
  • {objectId} - number which increasing every iteration in 1
  • {Description1}- we have a file with ~ 70K lines so in
    each iteration the Load Runner choose 1 random line for description
  • {title} - the same as description - but another file for
    title
  • {Status} - rnadom staus (overall there are 3 statuses
  • {First_RndNumber} - random nuber between 1 to 800
  • The all other parameters contain random number

Search cases:
you can see in the attached file(SearchCases.txt) the search
operation, please let me know if you need clarification for this action.

Workload:
Create cases was executed with 20 concurrent users which create
case every ~ 4 seconds
Search cases was executed with 10 concurrent users which executed
the search every ~25 seconds.
The create and the search run on the same time on the system.

Thanks,
Shay

On Wed, Jun 2, 2010 at 12:29 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This certainly does not look good. I have just run a test against
0.8 with 2k TPS index and search operations (simple ones) and the open files
system seems to be good (constant around 80) (with an FS gateway).

Maybe you uncovered a problem here that I can't recreate. Can you
explain more about the test that you run? Go into details regarding the
operations that you do. Which search requests, what parameters? Do you use
scroll search request?

Maybe you can create a simple test case that I can run here?

-shay.banon

On Wed, Jun 2, 2010 at 11:16 AM, shay katz katzshay1@gmail.comwrote:

Hi,
I executed a new perfromance test for the Elasticsearch, and
have the same results.
Attached please find 2 files:

  1. Elastic_Search_NUMBER_OF_OPEN_FILES.xlsx - Describe the trend
    of the number of open files (lsof | wc -l is running on the
    system every 1 minute).
    btw
    Please look in the following output:
    [root]# *lsof | wc -l
    *23917
    [root]# *lsof | grep elastic | wc -l
    *23376
    2.The second file (ElasticSearchOpenFiles.zip) contain the output
    from the following command: lsof | grep elastic

Thanks,
Shay

On Tue, Jun 1, 2010 at 8:41 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

This should not happen. Is it still running? Can you mail me the
a sample of the lsof output?

On Tue, Jun 1, 2010 at 2:23 PM, Shay katzshay1@gmail.comwrote:

Hi,
I'm conducting a test for create and search indexes(5 case
creation
every 1 seconds and 1 search every ~ 2 seconds.
I monitored the number of open files every 1 minute (lsof | wc
-l) and
it seems there is a gradual growth in the number of open files
though
there is no increasing in the load over the time.
After ~ 20 hours 17K open files were on the system, 15K of them
belong
to the Elasticsearch.
Is it a noraml behaior of the Elasticsearch?
I have a file with the list of the open files and excel which
shows
the trend of the number of open files on the system, so if you
would
like to see it, please send me a mail and i'll send you these
files.
Thanks,
Shay