We have multiple fields in our record, but single field is ngram analyzed
on which we search results for. This search need to performed on all 5
shards of the host to get results, as there is no routing in our case.
We observed huge variations in search TPS with decrease in elastic search
heap memory size. Attached bigdesk images for both of the below cases!
CASE1)
When ess_heap size = 16gb
Search tps observed is 50
CASE2)
When ess_heap_size=30gb
Search tps observed is 18
Surprising thing is as we decrease ess_heap_size the search tps got
increased. All the resources(cpu,memory etc) are not fully utilized, OS
heap memory is not changed much, observed lot of zig zag ess_heap
usage(increase and decrease of ess heap usage, may be because of high index
size that need to be brought to RAM) and reads I/Os followed same zig zag
manner in both the cases.
Please note that we have run this experiment multiple times and observed
the same pattern. Can you please guide on what is going wrong? what dec
ess_heap increasing the tps, should we further decrease to achieve better
tps or we doing something wrong?
So you believe in "the more heap the better"? This assumption you have just
proved wrong. Note, in your test, at 16GB heap, GC took half a second,
while at 30GB heap, GC took around a second (and maybe more overhead in
stop-the-world pauses). This is a hint about current JVM GC scalability of
Java 7.
We have multiple fields in our record, but single field is ngram analyzed
on which we search results for. This search need to performed on all 5
shards of the host to get results, as there is no routing in our case.
We observed huge variations in search TPS with decrease in elastic
search heap memory size. Attached bigdesk images for both of the below
cases!
CASE1)
When ess_heap size = 16gb
Search tps observed is 50
CASE2)
When ess_heap_size=30gb
Search tps observed is 18
Surprising thing is as we decrease ess_heap_size the search tps got
increased. All the resources(cpu,memory etc) are not fully utilized, OS
heap memory is not changed much, observed lot of zig zag ess_heap
usage(increase and decrease of ess heap usage, may be because of high index
size that need to be brought to RAM) and reads I/Os followed same zig zag
manner in both the cases.
Please note that we have run this experiment multiple times and
observed the same pattern. Can you please guide on what is going wrong?
what dec ess_heap increasing the tps, should we further decrease to achieve
better tps or we doing something wrong?
They are couple of questions pertaining to above graphs
Yes the GC time taken is doubled, but the (frequency)rate of GC cycle is
high(almost double/triple) in 16gb compared to 30gb.
Why complete memory is not utilized before cleaning up the memory, for
eg for 30gb, maximum usage is around 20gb, for 16gb it is 11gb.
This looks like thrashing!, is it because of large index size(76.6GB) on
single host ?,
PS: GC used here is G1GC
On Thursday, 19 February 2015 00:54:25 UTC+5:30, Jörg Prante wrote:
So you believe in "the more heap the better"? This assumption you have
just proved wrong. Note, in your test, at 16GB heap, GC took half a second,
while at 30GB heap, GC took around a second (and maybe more overhead in
stop-the-world pauses). This is a hint about current JVM GC scalability of
Java 7.
Jörg
On Wed, Feb 18, 2015 at 7:45 PM, Srikanth Valiveti <vsrikant...@gmail.com
<javascript:>> wrote:
We have multiple fields in our record, but single field is ngram analyzed
on which we search results for. This search need to performed on all 5
shards of the host to get results, as there is no routing in our case.
We observed huge variations in search TPS with decrease in elastic
search heap memory size. Attached bigdesk images for both of the below
cases!
CASE1)
When ess_heap size = 16gb
Search tps observed is 50
CASE2)
When ess_heap_size=30gb
Search tps observed is 18
Surprising thing is as we decrease ess_heap_size the search tps got
increased. All the resources(cpu,memory etc) are not fully utilized, OS
heap memory is not changed much, observed lot of zig zag ess_heap
usage(increase and decrease of ess heap usage, may be because of high index
size that need to be brought to RAM) and reads I/Os followed same zig zag
manner in both the cases.
Please note that we have run this experiment multiple times and
observed the same pattern. Can you please guide on what is going wrong?
what dec ess_heap increasing the tps, should we further decrease to achieve
better tps or we doing something wrong?
Smaller JVM heap means more free RAM for the OS to cache hot pages from
your index ... in general you should only give the JVM as much as it needs
(will ever need) and a bit more for safety, and give the rest to the OS so
it can put hot parts of your index in RAM.
They are couple of questions pertaining to above graphs
Yes the GC time taken is doubled, but the (frequency)rate of GC cycle
is high(almost double/triple) in 16gb compared to 30gb.
Why complete memory is not utilized before cleaning up the memory, for
eg for 30gb, maximum usage is around 20gb, for 16gb it is 11gb.
This looks like thrashing!, is it because of large index size(76.6GB)
on single host ?,
PS: GC used here is G1GC
On Thursday, 19 February 2015 00:54:25 UTC+5:30, Jörg Prante wrote:
So you believe in "the more heap the better"? This assumption you have
just proved wrong. Note, in your test, at 16GB heap, GC took half a second,
while at 30GB heap, GC took around a second (and maybe more overhead in
stop-the-world pauses). This is a hint about current JVM GC scalability of
Java 7.
We have multiple fields in our record, but single field is ngram
analyzed on which we search results for. This search need to performed on
all 5 shards of the host to get results, as there is no routing in our case.
We observed huge variations in search TPS with decrease in elastic
search heap memory size. Attached bigdesk images for both of the below
cases!
CASE1)
When ess_heap size = 16gb
Search tps observed is 50
CASE2)
When ess_heap_size=30gb
Search tps observed is 18
Surprising thing is as we decrease ess_heap_size the search tps got
increased. All the resources(cpu,memory etc) are not fully utilized, OS
heap memory is not changed much, observed lot of zig zag ess_heap
usage(increase and decrease of ess heap usage, may be because of high index
size that need to be brought to RAM) and reads I/Os followed same zig zag
manner in both the cases.
Please note that we have run this experiment multiple times and
observed the same pattern. Can you please guide on what is going wrong?
what dec ess_heap increasing the tps, should we further decrease to achieve
better tps or we doing something wrong?
Smaller JVM heap means more free RAM for the OS to cache hot pages from
your index ... in general you should only give the JVM as much as it needs
(will ever need) and a bit more for safety, and give the rest to the OS so
it can put hot parts of your index in RAM.
They are couple of questions pertaining to above graphs
Yes the GC time taken is doubled, but the (frequency)rate of GC cycle
is high(almost double/triple) in 16gb compared to 30gb.
Why complete memory is not utilized before cleaning up the memory, for
eg for 30gb, maximum usage is around 20gb, for 16gb it is 11gb.
This looks like thrashing!, is it because of large index size(76.6GB)
on single host ?,
PS: GC used here is G1GC
On Thursday, 19 February 2015 00:54:25 UTC+5:30, Jörg Prante wrote:
So you believe in "the more heap the better"? This assumption you have
just proved wrong. Note, in your test, at 16GB heap, GC took half a second,
while at 30GB heap, GC took around a second (and maybe more overhead in
stop-the-world pauses). This is a hint about current JVM GC scalability of
Java 7.
We have multiple fields in our record, but single field is ngram
analyzed on which we search results for. This search need to performed on
all 5 shards of the host to get results, as there is no routing in our case.
We observed huge variations in search TPS with decrease in elastic
search heap memory size. Attached bigdesk images for both of the below
cases!
CASE1)
When ess_heap size = 16gb
Search tps observed is 50
CASE2)
When ess_heap_size=30gb
Search tps observed is 18
Surprising thing is as we decrease ess_heap_size the search tps got
increased. All the resources(cpu,memory etc) are not fully utilized, OS
heap memory is not changed much, observed lot of zig zag ess_heap
usage(increase and decrease of ess heap usage, may be because of high index
size that need to be brought to RAM) and reads I/Os followed same zig zag
manner in both the cases.
Please note that we have run this experiment multiple times and
observed the same pattern. Can you please guide on what is going wrong?
what dec ess_heap increasing the tps, should we further decrease to achieve
better tps or we doing something wrong?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.