Too many open files


(Alex-2) #1

Hi Shay

after replacing the .war file and restarting the server, I get the
following error: https://gist.github.com/1340424
I'm using v. 0.18.2 with aws-s3. I never got such error with previous
versions. The exception is thrown when the first user accesses the
servlet.
What's going on?

Thanks!

Cheers
Alex


(Huy Le) #2

Not sure if v. 0.18.2 is using more file descriptor. We configured
our with 80,000 file descriptor. It was running fine with 0.17.x.
Upgraded to 0.18.2 5 days ago, and just got too open file error
today. Just bumped limit to 140,000.

Huy

On Nov 4, 5:07 pm, Alex alessandro.ussegliovire...@gmail.com wrote:

Hi Shay

after replacing the .war file and restarting the server, I get the
following error:https://gist.github.com/1340424
I'm using v. 0.18.2 with aws-s3. I never got such error with previous
versions. The exception is thrown when the first user accesses the
servlet.
What's going on?

Thanks!

Cheers
Alex


(Shay Banon) #3

0.18.x should not use more file descriptors than 0.17. Can you tell which
file descriptors are being used by the process (lsof -p), or, if anything
else changed except for the version bump?

On Fri, Nov 4, 2011 at 11:23 PM, Huy Le huyle@leveragingit.com wrote:

Not sure if v. 0.18.2 is using more file descriptor. We configured
our with 80,000 file descriptor. It was running fine with 0.17.x.
Upgraded to 0.18.2 5 days ago, and just got too open file error
today. Just bumped limit to 140,000.

Huy

On Nov 4, 5:07 pm, Alex alessandro.ussegliovire...@gmail.com wrote:

Hi Shay

after replacing the .war file and restarting the server, I get the
following error:https://gist.github.com/1340424
I'm using v. 0.18.2 with aws-s3. I never got such error with previous
versions. The exception is thrown when the first user accesses the
servlet.
What's going on?

Thanks!

Cheers
Alex


(Huy Le) #4

In our case, only configuration/application change is es version
change. However, we do have higher traffic load than we used to.
lsof shows almost all of the file descriptors used were on files in ES
data directory (nodes/0/indices). Here is outrcluster info:

{
"active_primary_shards": 60,
"active_shards": 180,
"cluster_name": "MyCluster",
"initializing_shards": 0,
"number_of_data_nodes": 6,
"number_of_nodes": 6,
"relocating_shards": 0,
"status": "green",
"timed_out": false,
"unassigned_shards": 0
}

This is on a cluster with 2 indices and 2 replicas.

Huy

On Nov 7, 10:45 pm, Shay Banon kim...@gmail.com wrote:

0.18.x should not use more file descriptors than 0.17. Can you tell which
file descriptors are being used by the process (lsof -p), or, if anything
else changed except for the version bump?

On Fri, Nov 4, 2011 at 11:23 PM, Huy Le hu...@leveragingit.com wrote:

Not sure if v. 0.18.2 is using more file descriptor. We configured
our with 80,000 file descriptor. It was running fine with 0.17.x.
Upgraded to 0.18.2 5 days ago, and just got too open file error
today. Just bumped limit to 140,000.

Huy

On Nov 4, 5:07 pm, Alex alessandro.ussegliovire...@gmail.com wrote:

Hi Shay

after replacing the .war file and restarting the server, I get the
following error:https://gist.github.com/1340424
I'm using v. 0.18.2 with aws-s3. I never got such error with previous
versions. The exception is thrown when the first user accesses the
servlet.
What's going on?

Thanks!

Cheers
Alex


(Huy Le) #5

At the time the issue occurred, there were 79501 files in data
directory nodes/0/indices/*. It seems like there were too many
segment files got created.

Huy

On Nov 8, 8:55 am, Huy Le hu...@leveragingit.com wrote:

In our case, only configuration/application change is es version
change. However, we do have higher traffic load than we used to.
lsof shows almost all of the file descriptors used were onfilesin ES
data directory (nodes/0/indices). Here is outrcluster info:

{
"active_primary_shards": 60,
"active_shards": 180,
"cluster_name": "MyCluster",
"initializing_shards": 0,
"number_of_data_nodes": 6,
"number_of_nodes": 6,
"relocating_shards": 0,
"status": "green",
"timed_out": false,
"unassigned_shards": 0

}

This is on a cluster with 2 indices and 2 replicas.

Huy

On Nov 7, 10:45 pm, Shay Banon kim...@gmail.com wrote:

0.18.x should not use more file descriptors than 0.17. Can you tell which
file descriptors are being used by the process (lsof -p), or, if anything
else changed except for the version bump?

On Fri, Nov 4, 2011 at 11:23 PM, Huy Le hu...@leveragingit.com wrote:

Not sure if v. 0.18.2 is using more file descriptor. We configured
our with 80,000 file descriptor. It was running fine with 0.17.x.
Upgraded to 0.18.2 5 days ago, and just gottooopen file error
today. Just bumped limit to 140,000.

Huy

On Nov 4, 5:07 pm, Alex alessandro.ussegliovire...@gmail.com wrote:

Hi Shay

after replacing the .war file and restarting the server, I get the
following error:https://gist.github.com/1340424
I'm using v. 0.18.2 with aws-s3. I never got such error with previous
versions. The exception is thrown when the first user accesses the
servlet.
What's going on?

Thanks!

Cheers
Alex


(system) #6