Elasticsearch node under utilising file descriptors

Elastic Stack Elasticsearch
8

You really need to set this "bootstrap.mlockall: true", it helps with
performance and stops the process being swapped out, which is extremely
expensive.
Also depending on the minor version of the 0.90 release, you can try
enabling compression, index.store.compress.stored and
index.store.compress.tv.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 19:29, Tamanjit Bindra bindra.tamanjit@gmail.com wrote:

Here http://pastebin.com/6EAQ8znF is the link to the config.

On Wednesday, May 14, 2014 2:41:41 PM UTC+5:30, Mark Walkom wrote:

The prime objectives you should be aiming for here are 1) update to
Oracle JDK (7u55), you will get much better performance and stability and,
2) upgrade ES to the latest, for the same reasons.

Can you gist/pastebin your current config? There be little that can be
done with such an old version, but we can take a look.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:37, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
Sorry about the paste thing.

I have recently joined the organisation and they have been using ES for
some time now. I am not too sure if there is any plan to migrate to 1.X
version any time soon. Although yes that is something that we may have to
do eventually. But is there any possibility of tweaking in the current
scenario?

Java version details:

java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

On Wednesday, May 14, 2014 11:49:19 AM UTC+5:30, Mark Walkom wrote:

It's better to put that sort of thing in a gist/pastebin :slight_smile:

Any reason you aren't running a more recent version of ES? You'll get
much better performance from the new versions.
Also can you be specific about your java, what update, OpenJDK or
Oracle?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:17, Tamanjit Bindra bindra....@gmail.com wrote:

Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7

The stats are as under :

"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}

I am attaching along the hot_threads output, though I couldn't make
much sense of it. Pardon me if I sound N00Bish, because I am pretty new to
ES.

On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:

ES will use descriptors to open files it needs for general operations
and it can have a lot of those open at one time. But the level you've set
and the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what
version of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads
(GET /_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a
better idea of what your resource usage is like on a cluster level. Bigdesk
is good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even
though the upper limit is 65535, ES is only using about 2K+ file
descriptors at any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated
to ES.

Please Help!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a
4-470b-96db-08117f554b7d%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/518692f6-e743-4323-bccf-44e299471c86%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/518692f6-e743-4323-bccf-44e299471c86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b-XQJGE0k_W-r4jc2QTWvTAKRSA_RK%3Di%3DBw4dmoOgJjA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.