ES load test ended up with out of memory error after enabling the clustering

Manjula_Piyumal · April 16, 2015, 3:43pm

Hi all,

I am trying to run load test with ES to identify system requirements and
optimum configurations with respect to my load. I have 10 data publishing
tasks and 100 data consuming tasks in my load test.
Data publisher : Each publisher publishes data in every minute and it
publishes 1700 records as a batch using java bulk API.
Data consumer : Each consumer runs in every minute and run a query with
randomly selected aggregation type(average, minimum or maximum) for a
selected data for last hour.
example query that consumer run in every minute :

SearchResponse searchResponse =
client.prepareSearch("myIndex").setTypes("myRecordType")
.setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchQuery("filed1",
"value1"),FilterBuilders.rangeFilter("field2").from("value2").to("value3")))
.addAggregation(AggregationBuilders.avg("AVG_NAME").field("field3")).execute().actionGet();

I have run above test case in my local machine without ES clustering and it
was run around 4 hours without any errors. Memory consumption of ES was
under 2GB. After that I have run same test case in three node ES
cluster(EC2 instances) and ES has ended up with out of memory error after
around 5 minutes in that case. My all three instances have following same
hardware configurations,

8GB RAM
80GB SSD hard disk
4 core CPU

Instance 1
Elasticsearch server (4GB heap)
10 data publishers which will publish data to the local ES server

Instance 2
Elasticsearch server (8GB heap)
10 consumers which will query data from the local ES server

Instance 3
Elasticsearch server (4GB heap)

I'm using ES 1.5.1 version with jdk 1.8.0_40.

My ES cluster have following custom configurations (all other
configurations are default configurations)

bootstrap.mlockall: true
indices.fielddata.cache.size: "30%"
indices.cache.filter.size: "30%
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1:9300","host2:9300","host3:9300"]

I believe I have missed something here regarding ES clustering
configuration. Please help me to identify what I have missed here. I want
to reduce the memory utilization as much as possible, that's why I have
given only 4GB heap to ES. If there is a way to reduce the memory
consumption by reducing read consistency level that option is also OK for
me. I have increased the refresh interval for my index, but still no luck

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f8ebc90-dd37-4da7-97bd-3c0d1c00165c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · April 16, 2015, 4:37pm

Did you assign different heap sizes? Please use same heap size for all data
nodes. Do not limit cache to 30%, this is very small. Let ES use the
default settings.

Jörg

On Thu, Apr 16, 2015 at 5:43 PM, Manjula Piyumal manjulapiyumal@gmail.com
wrote:

Hi all,

I am trying to run load test with ES to identify system requirements and
optimum configurations with respect to my load. I have 10 data publishing
tasks and 100 data consuming tasks in my load test.
Data publisher : Each publisher publishes data in every minute and it
publishes 1700 records as a batch using java bulk API.
Data consumer : Each consumer runs in every minute and run a query with
randomly selected aggregation type(average, minimum or maximum) for a
selected data for last hour.
example query that consumer run in every minute :

SearchResponse searchResponse =
client.prepareSearch("myIndex").setTypes("myRecordType")
.setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchQuery("filed1",
"value1"),FilterBuilders.rangeFilter("field2").from("value2").to("value3")))

.addAggregation(AggregationBuilders.avg("AVG_NAME").field("field3")).execute().actionGet();

I have run above test case in my local machine without ES clustering and
it was run around 4 hours without any errors. Memory consumption of ES was
under 2GB. After that I have run same test case in three node ES
cluster(EC2 instances) and ES has ended up with out of memory error after
around 5 minutes in that case. My all three instances have following same
hardware configurations,

8GB RAM
80GB SSD hard disk
4 core CPU

Instance 1
Elasticsearch server (4GB heap)
10 data publishers which will publish data to the local ES server

Instance 2
Elasticsearch server (8GB heap)
10 consumers which will query data from the local ES server

Instance 3
Elasticsearch server (4GB heap)

I'm using ES 1.5.1 version with jdk 1.8.0_40.

My ES cluster have following custom configurations (all other
configurations are default configurations)

bootstrap.mlockall: true
indices.fielddata.cache.size: "30%"
indices.cache.filter.size: "30%
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1:9300","host2:9300","host3:9300"]

I believe I have missed something here regarding ES clustering
configuration. Please help me to identify what I have missed here. I want
to reduce the memory utilization as much as possible, that's why I have
given only 4GB heap to ES. If there is a way to reduce the memory
consumption by reducing read consistency level that option is also OK for
me. I have increased the refresh interval for my index, but still no luck

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9f8ebc90-dd37-4da7-97bd-3c0d1c00165c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9f8ebc90-dd37-4da7-97bd-3c0d1c00165c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGuT%3DEo74poKcyVhfgiW5W82AFmMSX9ouUEJDAPqhajpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Manjula_Piyumal · April 16, 2015, 6:05pm

Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES servers.
I have tried without limiting the cache size as the first attempt. But it
also got the out of memory error. Am I not missing any other configuration?
Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · April 16, 2015, 9:21pm

I have thousands of concurrent indexing/queries running per second on
non-virtualized servers.

4G heap is ok, it is more than enough, there should be other reasons for
OOM I am sure.

Maybe Bigdesk can help for monitoring heap and cache eviction rates.

I think I can not help any more - no experience with EC2.

Jörg

On Thu, Apr 16, 2015 at 8:05 PM, Manjula Piyumal manjulapiyumal@gmail.com
wrote:

Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES servers.
I have tried without limiting the cache size as the first attempt. But it
also got the out of memory error. Am I not missing any other configuration?
Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Manjula_Piyumal · April 17, 2015, 3:48am

Hi Jörg,

Thanks a lot for your help. May I know what is the size of one record that
you are publishing to ES. And did you do any configuration changes to the
default configurations?

Thanks
Manjula

On Fri, Apr 17, 2015 at 2:51 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I have thousands of concurrent indexing/queries running per second on
non-virtualized servers.

4G heap is ok, it is more than enough, there should be other reasons for
OOM I am sure.

Maybe Bigdesk can help for monitoring heap and cache eviction rates.

I think I can not help any more - no experience with EC2.

Jörg

On Thu, Apr 16, 2015 at 8:05 PM, Manjula Piyumal <manjulapiyumal@gmail.com

wrote:

Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES
servers. I have tried without limiting the cache size as the first attempt.
But it also got the out of memory error. Am I not missing any other
configuration? Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/ci_YFAPLYX4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFLAp0Wnsw8ujLV0Bz8vi7tswwZkcSb9HHN1LR17JnNV_5B%2BVA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Manjula_Piyumal · April 25, 2015, 6:12pm

Hi Jörg,

You are correct, there was a problem in my query client which produced the
query load than expected. Thanks again for your help and sorry for the
inconvenience.

Thanks
Manjula

On Fri, Apr 17, 2015 at 9:18 AM, Manjula Piyumal manjulapiyumal@gmail.com
wrote:

Hi Jörg,

Thanks a lot for your help. May I know what is the size of one record that
you are publishing to ES. And did you do any configuration changes to the
default configurations?

Thanks
Manjula

On Fri, Apr 17, 2015 at 2:51 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I have thousands of concurrent indexing/queries running per second on
non-virtualized servers.

4G heap is ok, it is more than enough, there should be other reasons for
OOM I am sure.

Maybe Bigdesk can help for monitoring heap and cache eviction rates.

I think I can not help any more - no experience with EC2.

Jörg

On Thu, Apr 16, 2015 at 8:05 PM, Manjula Piyumal <
manjulapiyumal@gmail.com> wrote:

Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES
servers. I have tried without limiting the cache size as the first attempt.
But it also got the out of memory error. Am I not missing any other
configuration? Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/ci_YFAPLYX4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFLAp0Vo5iQZV%3D7cBGp3WisLuO-2OLM18-OGDyJ9QA4jCxByfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

jprante · April 25, 2015, 6:30pm

Glad you solved it. You're welcome.

Best,

Jörg

On Sat, Apr 25, 2015 at 8:12 PM, Manjula Piyumal manjulapiyumal@gmail.com
wrote:

Hi Jörg,

You are correct, there was a problem in my query client which produced the
query load than expected. Thanks again for your help and sorry for the
inconvenience.

Thanks
Manjula

On Fri, Apr 17, 2015 at 9:18 AM, Manjula Piyumal <manjulapiyumal@gmail.com

wrote:

Hi Jörg,

Thanks a lot for your help. May I know what is the size of one record
that you are publishing to ES. And did you do any configuration changes to
the default configurations?

Thanks
Manjula

On Fri, Apr 17, 2015 at 2:51 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I have thousands of concurrent indexing/queries running per second on
non-virtualized servers.

4G heap is ok, it is more than enough, there should be other reasons for
OOM I am sure.

Maybe Bigdesk can help for monitoring heap and cache eviction rates.

I think I can not help any more - no experience with EC2.

Jörg

On Thu, Apr 16, 2015 at 8:05 PM, Manjula Piyumal <
manjulapiyumal@gmail.com> wrote:

Hi Jörg,

Sorry, my bad. It's a typo, I have used 4GB heap for all three ES
servers. I have tried without limiting the cache size as the first attempt.
But it also got the out of memory error. Am I not missing any other
configuration? Or is this load is too much for 4GB heap?

Thanks
Manjula

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c41b0dd9-31b8-45ee-909e-0f5e2995c4ac%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/ci_YFAPLYX4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEF0pFDGWHiUjBYShTSMBmk%3D-zMW8mkZQMqNpMO74sNPA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAFLAp0Vo5iQZV%3D7cBGp3WisLuO-2OLM18-OGDyJ9QA4jCxByfA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAFLAp0Vo5iQZV%3D7cBGp3WisLuO-2OLM18-OGDyJ9QA4jCxByfA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDJgqGyUOVA6T14ryfsh-Mmj5PoyzcXqWSSYS7zubsAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.