Number of open files using ES 0.90.0.RC1

Elastic Stack Elasticsearch
1

Hi there,

I'm having some issues with # open files on one of our test instances here.

I have been running 2 instances (0.20.4 and 0.90.0.RC1) for the past few
days without incident to do some comparison testing. Note all the instances
are separate single node clusters(not multiple nodes on the same box) and
the total data amount per instance is < 0.5m documents spread over 3
indexes with 1-2 types per index.

I then wanted to migrate my data from the existing 0.90.0.RC1 instance to
another new 90RC1 instance for testing (migrating data from 1 schema to
another via a new mapping) and was getting a "red" status on the cluster
health for the new instance.

When i drilled down into the stats (some links point out that number of
open files could be the issue), i noticed that the number of open files
were very high. I shut down all the instances and started up the original
0.90.0.RC1 instance:

Some info here:

~/elasticsearch-0.90.0.RC1# ulimit -n
64000

Even though ES sees the limit of 64000 (verified via curl), it is reporting
that there is 937 open file descriptors whereas doing lsof shows ~54000.
Also, the numbers seem very strange (106 usages of certain files, 53 of
others).

For reference, this is running on Ubuntu 12.10 with Java 1.6 b37

~/elasticsearch-0.90.0.RC1# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.10
DISTRIB_CODENAME=quantal
DISTRIB_DESCRIPTION="Ubuntu 12.10"
:~/elasticsearch-0.90.0.RC1# java -version
java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)

Running using /bin/elasticsearch in background mode.

So i'm really wondering:
a) Why is the usage so high? On the 0.20.4 instance, the number of open
files is greatly lower ~5000 vs 50000 for a similar dataset)
b) How come the number reported by the API stats command is different to
LSOF?
c) Should i change my segments/merge stats as has been suggested in some
other posts for open file issues?

Any help/tips greatly appreciated.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi,

That 937 is close to "954 (ESTABLISHED)" from lsof.
What are other open files beyond head -100? I'm guessing you'll see a bunch
of index files. Try optimizing and then check again.

Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Tuesday, March 26, 2013 4:51:45 AM UTC-4, Derry O' Sullivan wrote:

Hi there,

I'm having some issues with # open files on one of our test instances here.

I have been running 2 instances (0.20.4 and 0.90.0.RC1) for the past few
days without incident to do some comparison testing. Note all the instances
are separate single node clusters(not multiple nodes on the same box) and
the total data amount per instance is < 0.5m documents spread over 3
indexes with 1-2 types per index.

I then wanted to migrate my data from the existing 0.90.0.RC1 instance to
another new 90RC1 instance for testing (migrating data from 1 schema to
another via a new mapping) and was getting a "red" status on the cluster
health for the new instance.

When i drilled down into the stats (some links point out that number of
open files could be the issue), i noticed that the number of open files
were very high. I shut down all the instances and started up the original
0.90.0.RC1 instance:

Some info here:
Open files in ES 0.90.RC1 · GitHub

~/elasticsearch-0.90.0.RC1# ulimit -n
64000

Even though ES sees the limit of 64000 (verified via curl), it is
reporting that there is 937 open file descriptors whereas doing lsof shows
~54000. Also, the numbers seem very strange (106 usages of certain files,
53 of others).

For reference, this is running on Ubuntu 12.10 with Java 1.6 b37

~/elasticsearch-0.90.0.RC1# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.10
DISTRIB_CODENAME=quantal
DISTRIB_DESCRIPTION="Ubuntu 12.10"
:~/elasticsearch-0.90.0.RC1# java -version
java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)

Running using /bin/elasticsearch in background mode.

So i'm really wondering:
a) Why is the usage so high? On the 0.20.4 instance, the number of open
files is greatly lower ~5000 vs 50000 for a similar dataset)
b) How come the number reported by the API stats command is different to
LSOF?
c) Should i change my segments/merge stats as has been suggested in some
other posts for open file issues?

Any help/tips greatly appreciated.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.