Elasticsearch node under utilising file descriptors


(Tamanjit Bindra) #1

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even though the
upper limit is 65535, ES is only using about 2K+ file descriptors at any
given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated to ES.

Please Help!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

ES will use descriptors to open files it needs for general operations and
it can have a lot of those open at one time. But the level you've set and
the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what version
of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads (GET
/_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a better
idea of what your resource usage is like on a cluster level. Bigdesk is
good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra bindra.tamanjit@gmail.com wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even though
the upper limit is 65535, ES is only using about 2K+ file descriptors at
any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated to ES.

Please Help!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Z%3DA8FRq-d4rW9nhBsmkx4KDUS0Wyp9Vu%2BzQKM4oPeCxw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Tamanjit Bindra) #3

Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7

The stats are as under :

"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}

I am attaching along the hot_threads output, though I couldn't make much
sense of it. Pardon me if I sound N00Bish, because I am pretty new to ES.

On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:

ES will use descriptors to open files it needs for general operations and
it can have a lot of those open at one time. But the level you've set and
the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what
version of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads (GET
/_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a better
idea of what your resource usage is like on a cluster level. Bigdesk is
good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra <bindra....@gmail.com <javascript:>>wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even though
the upper limit is 65535, ES is only using about 2K+ file descriptors at
any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated to ES.

Please Help!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #4

It's better to put that sort of thing in a gist/pastebin :slight_smile:

Any reason you aren't running a more recent version of ES? You'll get much
better performance from the new versions.
Also can you be specific about your java, what update, OpenJDK or Oracle?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:17, Tamanjit Bindra bindra.tamanjit@gmail.com wrote:

Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7

The stats are as under :

"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}

I am attaching along the hot_threads output, though I couldn't make much
sense of it. Pardon me if I sound N00Bish, because I am pretty new to ES.

On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:

ES will use descriptors to open files it needs for general operations and
it can have a lot of those open at one time. But the level you've set and
the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what
version of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads (GET
/_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a
better idea of what your resource usage is like on a cluster level. Bigdesk
is good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even though
the upper limit is 65535, ES is only using about 2K+ file descriptors at
any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated to ES.

Please Help!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aHa485CQOcu1Mk8m6_w115-0VwKofT9CD%2BLyLhgue%2BVg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Tamanjit Bindra) #5

Hi,
Sorry about the paste thing.

I have recently joined the organisation and they have been using ES for
some time now. I am not too sure if there is any plan to migrate to 1.X
version any time soon. Although yes that is something that we may have to
do eventually. But is there any possibility of tweaking in the current
scenario?

Java version details:

java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

On Wednesday, May 14, 2014 11:49:19 AM UTC+5:30, Mark Walkom wrote:

It's better to put that sort of thing in a gist/pastebin :slight_smile:

Any reason you aren't running a more recent version of ES? You'll get much
better performance from the new versions.
Also can you be specific about your java, what update, OpenJDK or Oracle?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 14 May 2014 16:17, Tamanjit Bindra <bindra....@gmail.com <javascript:>>wrote:

Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7

The stats are as under :

"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}

I am attaching along the hot_threads output, though I couldn't make much
sense of it. Pardon me if I sound N00Bish, because I am pretty new to ES.

On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:

ES will use descriptors to open files it needs for general operations
and it can have a lot of those open at one time. But the level you've set
and the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what
version of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads
(GET /_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a
better idea of what your resource usage is like on a cluster level. Bigdesk
is good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even though
the upper limit is 65535, ES is only using about 2K+ file descriptors at
any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated to
ES.

Please Help!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #6

The prime objectives you should be aiming for here are 1) update to Oracle
JDK (7u55), you will get much better performance and stability and, 2)
upgrade ES to the latest, for the same reasons.

Can you gist/pastebin your current config? There be little that can be done
with such an old version, but we can take a look.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:37, Tamanjit Bindra bindra.tamanjit@gmail.com wrote:

Hi,
Sorry about the paste thing.

I have recently joined the organisation and they have been using ES for
some time now. I am not too sure if there is any plan to migrate to 1.X
version any time soon. Although yes that is something that we may have to
do eventually. But is there any possibility of tweaking in the current
scenario?

Java version details:

java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

On Wednesday, May 14, 2014 11:49:19 AM UTC+5:30, Mark Walkom wrote:

It's better to put that sort of thing in a gist/pastebin :slight_smile:

Any reason you aren't running a more recent version of ES? You'll get
much better performance from the new versions.
Also can you be specific about your java, what update, OpenJDK or Oracle?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:17, Tamanjit Bindra bindra....@gmail.com wrote:

Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7

The stats are as under :

"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}

I am attaching along the hot_threads output, though I couldn't make much
sense of it. Pardon me if I sound N00Bish, because I am pretty new to ES.

On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:

ES will use descriptors to open files it needs for general operations
and it can have a lot of those open at one time. But the level you've set
and the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what
version of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads
(GET /_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a
better idea of what your resource usage is like on a cluster level. Bigdesk
is good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even
though the upper limit is 65535, ES is only using about 2K+ file
descriptors at any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated to
ES.

Please Help!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YsSgQM5w-in5fdYvGn6d7EhWLa-s0RgQH3TrR%3DxaTG-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Tamanjit Bindra) #7

Here http://pastebin.com/6EAQ8znF is the link to the config.

On Wednesday, May 14, 2014 2:41:41 PM UTC+5:30, Mark Walkom wrote:

The prime objectives you should be aiming for here are 1) update to Oracle
JDK (7u55), you will get much better performance and stability and, 2)
upgrade ES to the latest, for the same reasons.

Can you gist/pastebin your current config? There be little that can be
done with such an old version, but we can take a look.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 14 May 2014 16:37, Tamanjit Bindra <bindra....@gmail.com <javascript:>>wrote:

Hi,
Sorry about the paste thing.

I have recently joined the organisation and they have been using ES for
some time now. I am not too sure if there is any plan to migrate to 1.X
version any time soon. Although yes that is something that we may have to
do eventually. But is there any possibility of tweaking in the current
scenario?

Java version details:

java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

On Wednesday, May 14, 2014 11:49:19 AM UTC+5:30, Mark Walkom wrote:

It's better to put that sort of thing in a gist/pastebin :slight_smile:

Any reason you aren't running a more recent version of ES? You'll get
much better performance from the new versions.
Also can you be specific about your java, what update, OpenJDK or Oracle?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:17, Tamanjit Bindra bindra....@gmail.com wrote:

Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7

The stats are as under :

"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}

I am attaching along the hot_threads output, though I couldn't make
much sense of it. Pardon me if I sound N00Bish, because I am pretty new to
ES.

On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:

ES will use descriptors to open files it needs for general operations
and it can have a lot of those open at one time. But the level you've set
and the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what
version of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads
(GET /_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a
better idea of what your resource usage is like on a cluster level. Bigdesk
is good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even
though the upper limit is 65535, ES is only using about 2K+ file
descriptors at any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated to
ES.

Please Help!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/518692f6-e743-4323-bccf-44e299471c86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #8

You really need to set this "bootstrap.mlockall: true", it helps with
performance and stops the process being swapped out, which is extremely
expensive.
Also depending on the minor version of the 0.90 release, you can try
enabling compression, index.store.compress.stored and
index.store.compress.tv.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 19:29, Tamanjit Bindra bindra.tamanjit@gmail.com wrote:

Here http://pastebin.com/6EAQ8znF is the link to the config.

On Wednesday, May 14, 2014 2:41:41 PM UTC+5:30, Mark Walkom wrote:

The prime objectives you should be aiming for here are 1) update to
Oracle JDK (7u55), you will get much better performance and stability and,
2) upgrade ES to the latest, for the same reasons.

Can you gist/pastebin your current config? There be little that can be
done with such an old version, but we can take a look.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:37, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
Sorry about the paste thing.

I have recently joined the organisation and they have been using ES for
some time now. I am not too sure if there is any plan to migrate to 1.X
version any time soon. Although yes that is something that we may have to
do eventually. But is there any possibility of tweaking in the current
scenario?

Java version details:

java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

On Wednesday, May 14, 2014 11:49:19 AM UTC+5:30, Mark Walkom wrote:

It's better to put that sort of thing in a gist/pastebin :slight_smile:

Any reason you aren't running a more recent version of ES? You'll get
much better performance from the new versions.
Also can you be specific about your java, what update, OpenJDK or
Oracle?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:17, Tamanjit Bindra bindra....@gmail.com wrote:

Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7

The stats are as under :

"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}

I am attaching along the hot_threads output, though I couldn't make
much sense of it. Pardon me if I sound N00Bish, because I am pretty new to
ES.

On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:

ES will use descriptors to open files it needs for general operations
and it can have a lot of those open at one time. But the level you've set
and the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what
version of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads
(GET /_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a
better idea of what your resource usage is like on a cluster level. Bigdesk
is good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even
though the upper limit is 65535, ES is only using about 2K+ file
descriptors at any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated
to ES.

Please Help!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a
4-470b-96db-08117f554b7d%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/518692f6-e743-4323-bccf-44e299471c86%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/518692f6-e743-4323-bccf-44e299471c86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b-XQJGE0k_W-r4jc2QTWvTAKRSA_RK%3Di%3DBw4dmoOgJjA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Tamanjit Bindra) #9

I too had drilled down to mlockall. That surely needs a change. The rest
two am not sure. If I remember correctly 0.90 version already handles this.
Would need to check this.
One more thing. Is there a way to reconfigure mlockall without restarting a
node?
On 14 May 2014 16:10, "Mark Walkom" markw@campaignmonitor.com wrote:

You really need to set this "bootstrap.mlockall: true", it helps with
performance and stops the process being swapped out, which is extremely
expensive.
Also depending on the minor version of the 0.90 release, you can try
enabling compression, index.store.compress.stored and
index.store.compress.tv.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 19:29, Tamanjit Bindra bindra.tamanjit@gmail.com wrote:

Here http://pastebin.com/6EAQ8znF is the link to the config.

On Wednesday, May 14, 2014 2:41:41 PM UTC+5:30, Mark Walkom wrote:

The prime objectives you should be aiming for here are 1) update to
Oracle JDK (7u55), you will get much better performance and stability and,
2) upgrade ES to the latest, for the same reasons.

Can you gist/pastebin your current config? There be little that can be
done with such an old version, but we can take a look.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:37, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
Sorry about the paste thing.

I have recently joined the organisation and they have been using ES for
some time now. I am not too sure if there is any plan to migrate to 1.X
version any time soon. Although yes that is something that we may have to
do eventually. But is there any possibility of tweaking in the current
scenario?

Java version details:

java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

On Wednesday, May 14, 2014 11:49:19 AM UTC+5:30, Mark Walkom wrote:

It's better to put that sort of thing in a gist/pastebin :slight_smile:

Any reason you aren't running a more recent version of ES? You'll get
much better performance from the new versions.
Also can you be specific about your java, what update, OpenJDK or
Oracle?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:17, Tamanjit Bindra bindra....@gmail.com wrote:

Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7

The stats are as under :

"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}

I am attaching along the hot_threads output, though I couldn't make
much sense of it. Pardon me if I sound N00Bish, because I am pretty new to
ES.

On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:

ES will use descriptors to open files it needs for general
operations and it can have a lot of those open at one time. But the level
you've set and the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what
version of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads
(GET /_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a
better idea of what your resource usage is like on a cluster level. Bigdesk
is good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even
though the upper limit is 65535, ES is only using about 2K+ file
descriptors at any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated
to ES.

Please Help!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a
4-470b-96db-08117f554b7d%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/518692f6-e743-4323-bccf-44e299471c86%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/518692f6-e743-4323-bccf-44e299471c86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9ruH5k_rN_Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b-XQJGE0k_W-r4jc2QTWvTAKRSA_RK%3Di%3DBw4dmoOgJjA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624b-XQJGE0k_W-r4jc2QTWvTAKRSA_RK%3Di%3DBw4dmoOgJjA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADA%3D_UMa52BOvcXQnx709kgYkvgPsNOnet6ae80P7zHjQOiHfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #10

Later versions on 0.90 had compression enabled by default, but you haven't
supplied the exact version you are running so I took a guess.
You have to restart the node to apply mlockall.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 15 May 2014 02:27, Tamanjit Bindra bindra.tamanjit@gmail.com wrote:

I too had drilled down to mlockall. That surely needs a change. The rest
two am not sure. If I remember correctly 0.90 version already handles this.
Would need to check this.
One more thing. Is there a way to reconfigure mlockall without restarting
a node?
On 14 May 2014 16:10, "Mark Walkom" markw@campaignmonitor.com wrote:

You really need to set this "bootstrap.mlockall: true", it helps with
performance and stops the process being swapped out, which is extremely
expensive.
Also depending on the minor version of the 0.90 release, you can try
enabling compression, index.store.compress.stored and
index.store.compress.tv.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 19:29, Tamanjit Bindra bindra.tamanjit@gmail.com wrote:

Here http://pastebin.com/6EAQ8znF is the link to the config.

On Wednesday, May 14, 2014 2:41:41 PM UTC+5:30, Mark Walkom wrote:

The prime objectives you should be aiming for here are 1) update to
Oracle JDK (7u55), you will get much better performance and stability and,
2) upgrade ES to the latest, for the same reasons.

Can you gist/pastebin your current config? There be little that can be
done with such an old version, but we can take a look.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:37, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
Sorry about the paste thing.

I have recently joined the organisation and they have been using ES
for some time now. I am not too sure if there is any plan to migrate to 1.X
version any time soon. Although yes that is something that we may have to
do eventually. But is there any possibility of tweaking in the current
scenario?

Java version details:

java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

On Wednesday, May 14, 2014 11:49:19 AM UTC+5:30, Mark Walkom wrote:

It's better to put that sort of thing in a gist/pastebin :slight_smile:

Any reason you aren't running a more recent version of ES? You'll get
much better performance from the new versions.
Also can you be specific about your java, what update, OpenJDK or
Oracle?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 16:17, Tamanjit Bindra bindra....@gmail.com wrote:

Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7

The stats are as under :

"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}

I am attaching along the hot_threads output, though I couldn't make
much sense of it. Pardon me if I sound N00Bish, because I am pretty new to
ES.

On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:

ES will use descriptors to open files it needs for general
operations and it can have a lot of those open at one time. But the level
you've set and the level you're using are not related to your slow queries.

How much data do you have in your cluster? What version of ES, what
version of java? You should also check your slow log
(/var/log/elasticsearch/*_index_search_slowlog.log) and hot
threads (GET /_cluster/nodes/hot_threads) to see what is happening.
Also try installing a plugin like Marvel or ElasticHQ to give you a
better idea of what your resource usage is like on a cluster level. Bigdesk
is good but it limited to a node only view.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 14 May 2014 15:47, Tamanjit Bindra bindra....@gmail.com wrote:

Hi,
I have read herehttp://www.elasticsearch.org/tutorials/too-many-open-files/,
that we should set the soft and hard values of nooffiles for ES to utilise.

I have done that, but somehow my BigDesk console shows that even
though the upper limit is 65535, ES is only using about 2K+ file
descriptors at any given moment.

I have set the ulimit -Hn and ulimit -Sn to 32000.

Why is this so?

I ask this because my queries are performing slow.

We are using 2 nodes. Both servers have 16G RAM with 10G dedicated
to ES.

Please Help!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a
4-470b-96db-08117f554b7d%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a
0-4424-86b0-2322f18b6047%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d8317d44-f8d9-400f-b72d-089840db0cc7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/518692f6-e743-4323-bccf-44e299471c86%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/518692f6-e743-4323-bccf-44e299471c86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9ruH5k_rN_Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b-XQJGE0k_W-r4jc2QTWvTAKRSA_RK%3Di%3DBw4dmoOgJjA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624b-XQJGE0k_W-r4jc2QTWvTAKRSA_RK%3Di%3DBw4dmoOgJjA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CADA%3D_UMa52BOvcXQnx709kgYkvgPsNOnet6ae80P7zHjQOiHfQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CADA%3D_UMa52BOvcXQnx709kgYkvgPsNOnet6ae80P7zHjQOiHfQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624asEd%3DAHaOJnN9xYMdP%2BUGTEJL3L-TYTXS%2BKHfxtnKXZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #11