OutOfMemory

Elastic Stack Elasticsearch
1

Hi all,

We are using elasticsearch in our production backing dashboards with social media data. We are running 0.90 version.

Our volume of data grows every day, so we semi-regularly add servers to our cluster to keep things running smooth. A year ago we went from 4 nodes to 6 nodes. One month ago we have gone from 6 to 8 nodes. Only now, we are seeing OutOfMemory like issues again.

My question: It surprises me that we are hitting resource limits again so soon, increase in data does not explain that very well. Any suggestions for causes?

Best regards,
Eike dehling

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b7ca0070-84bc-4325-8a36-f8600289ae65%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

There is more to the issue than merely your configuration. What are your
queries? Are you doing a lot of aggregations, especially on on
high-cardinality fields. What kind of hardware are you running now?

Using the API, looks at your field cache usage. The field cache is held
within the Java heap space, which would cause the out of memory issues.

Cheers,

Ivan

On Thu, Oct 23, 2014 at 1:26 PM, Eike Dehling eike@buzzcapture.com wrote:

Hi all,

We are using elasticsearch in our production backing dashboards with
social media data. We are running 0.90 version.

Our volume of data grows every day, so we semi-regularly add servers to
our cluster to keep things running smooth. A year ago we went from 4 nodes
to 6 nodes. One month ago we have gone from 6 to 8 nodes. Only now, we are
seeing OutOfMemory like issues again.

My question: It surprises me that we are hitting resource limits again so
soon, increase in data does not explain that very well. Any suggestions for
causes?

Best regards,
Eike dehling

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b7ca0070-84bc-4325-8a36-f8600289ae65%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAnMdwVi-xq3oXEsYKROtcDwhR58pnZFsShLhmXvVg5VA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi ivan,

Thanks for the quick reply.

We are doing a lot of aggregations. And those are often the direct cause
for a crash: a particularly large aggregation using more than available
memory.

Field data is around 10gb, of the 31gb assigned to each node, field data
does not change much after our initial warmers are loaded.

I had expected that the recent addition of two nodes to the cluster would
give us enough resources to run trouble-free. Each node now has less shards
(from 6 nodes with 4 shards, to 8 nodes with 3 shards), and as such
searches and aggregates over one quarter less data? Is this simple math
true, or am i missing something?

How is field data expected to grow or shrink when going from 4 to 3 shards
on a node? Does it use about one quarter less memory?

Eike

On Thursday, October 23, 2014, Ivan Brusic ivan@brusic.com wrote:

There is more to the issue than merely your configuration. What are your
queries? Are you doing a lot of aggregations, especially on on
high-cardinality fields. What kind of hardware are you running now?

Using the API, looks at your field cache usage. The field cache is held
within the Java heap space, which would cause the out of memory issues.

Cheers,

Ivan

On Thu, Oct 23, 2014 at 1:26 PM, Eike Dehling <eike@buzzcapture.com
<javascript:_e(%7B%7D,'cvml','eike@buzzcapture.com');>> wrote:

Hi all,

We are using elasticsearch in our production backing dashboards with
social media data. We are running 0.90 version.

Our volume of data grows every day, so we semi-regularly add servers to
our cluster to keep things running smooth. A year ago we went from 4 nodes
to 6 nodes. One month ago we have gone from 6 to 8 nodes. Only now, we are
seeing OutOfMemory like issues again.

My question: It surprises me that we are hitting resource limits again so
soon, increase in data does not explain that very well. Any suggestions for
causes?

Best regards,
Eike dehling

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','elasticsearch%2Bunsubscribe@googlegroups.com');>
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b7ca0070-84bc-4325-8a36-f8600289ae65%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/w7aXHMa8Sls/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','elasticsearch%2Bunsubscribe@googlegroups.com');>
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAnMdwVi-xq3oXEsYKROtcDwhR58pnZFsShLhmXvVg5VA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAnMdwVi-xq3oXEsYKROtcDwhR58pnZFsShLhmXvVg5VA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Met vriendelijke groet,
Kind regards,

Eike Dehling
Lead Developer

Buzzcapture
Herengracht 180, 1016 BR, Amsterdam

T: +31 (0)20 3200377
M: +31 (0)6 45144840

LinkedIn | @buzzcapture

Recent
01-08-2014: Buzzcapture breidt uit! Bekijk de vacatures
https://www.buzzcapture.com/over-ons/werken-bij/op de site
30-07-2014: Tips voor je contentkalender
https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/
van
augustus en leestips
https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/
voor
op het strand
01-07-2014: EY en Tennet starten met Buzzcapture
28-06-2014: Ziekenhuizen LUMC en MUMC aan de slag met Buzzcapture
01-06-2014: Buzzcapture integreert print
https://www.buzzcapture.com/blog/buzzcapture-biedt-en-online-geintegreerde-monitoring-aan/
content
in Social Media Dashboard

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMp7VWTUk_iVpdFh9yNZWDFv-vz%3DWOW0m3ut-%2B2uXeL3OGgtFQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

It could tell more if you have the memory error. We are using version 0.90
as well and recently our traffic surge and our problem was due to the file
descriptor reach, based on the error in the log, so we increase the file
descriptor for the user, max number of processes per user.

This does not mean it would work for your situation but just saying you can
start to investigate from the log.

Hope it helps.

Jason

On Fri, Oct 24, 2014 at 2:00 AM, Eike Dehling eike@buzzcapture.com wrote:

Hi ivan,

Thanks for the quick reply.

We are doing a lot of aggregations. And those are often the direct cause
for a crash: a particularly large aggregation using more than available
memory.

Field data is around 10gb, of the 31gb assigned to each node, field data
does not change much after our initial warmers are loaded.

I had expected that the recent addition of two nodes to the cluster would
give us enough resources to run trouble-free. Each node now has less shards
(from 6 nodes with 4 shards, to 8 nodes with 3 shards), and as such
searches and aggregates over one quarter less data? Is this simple math
true, or am i missing something?

How is field data expected to grow or shrink when going from 4 to 3 shards
on a node? Does it use about one quarter less memory?

Eike

On Thursday, October 23, 2014, Ivan Brusic ivan@brusic.com wrote:

There is more to the issue than merely your configuration. What are your
queries? Are you doing a lot of aggregations, especially on on
high-cardinality fields. What kind of hardware are you running now?

Using the API, looks at your field cache usage. The field cache is held
within the Java heap space, which would cause the out of memory issues.

Cheers,

Ivan

On Thu, Oct 23, 2014 at 1:26 PM, Eike Dehling eike@buzzcapture.com
wrote:

Hi all,

We are using elasticsearch in our production backing dashboards with
social media data. We are running 0.90 version.

Our volume of data grows every day, so we semi-regularly add servers to
our cluster to keep things running smooth. A year ago we went from 4 nodes
to 6 nodes. One month ago we have gone from 6 to 8 nodes. Only now, we are
seeing OutOfMemory like issues again.

My question: It surprises me that we are hitting resource limits again
so soon, increase in data does not explain that very well. Any suggestions
for causes?

Best regards,
Eike dehling

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b7ca0070-84bc-4325-8a36-f8600289ae65%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/w7aXHMa8Sls/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAnMdwVi-xq3oXEsYKROtcDwhR58pnZFsShLhmXvVg5VA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAnMdwVi-xq3oXEsYKROtcDwhR58pnZFsShLhmXvVg5VA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Met vriendelijke groet,
Kind regards,

Eike Dehling
Lead Developer

Buzzcapture
Herengracht 180, 1016 BR, Amsterdam

T: +31 (0)20 3200377
M: +31 (0)6 45144840

LinkedIn | @buzzcapture

Recent
01-08-2014: Buzzcapture breidt uit! Bekijk de vacatures
https://www.buzzcapture.com/over-ons/werken-bij/op de site
30-07-2014: Tips voor je contentkalender
https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/ van
augustus en leestips
https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/ voor
op het strand
01-07-2014: EY en Tennet starten met Buzzcapture
28-06-2014: Ziekenhuizen LUMC en MUMC aan de slag met Buzzcapture
01-06-2014: Buzzcapture integreert print
https://www.buzzcapture.com/blog/buzzcapture-biedt-en-online-geintegreerde-monitoring-aan/ content
in Social Media Dashboard

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAMp7VWTUk_iVpdFh9yNZWDFv-vz%3DWOW0m3ut-%2B2uXeL3OGgtFQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAMp7VWTUk_iVpdFh9yNZWDFv-vz%3DWOW0m3ut-%2B2uXeL3OGgtFQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itw28qqSeDs5tDHQcROXX0TZLF6%2BY4k%3DXjQizRjPs4TR%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Thanks for the help.

I am getting errors like this, which i interpret as that execution of a
query requested more than available memory.

Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.FixedBitSet.(FixedBitSet.java:54)
at
org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:104)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:311)
...

What bothers me, is why our recent addition (last month) of two nodes (from
6 to 8) to the cluster has not solved this issue.

What i expect, is that growing the cluster from 6 to 8 nodes means each
node has about 25% more resources available. Less field data loaded per
node (since it now has less shards) and less data/shards to aggregate over.
I am looking for an explanation why this did not work that way, any hints
are welcome!

Eike

On Thu, Oct 23, 2014 at 8:11 PM, Jason Wee peichieh@gmail.com wrote:

It could tell more if you have the memory error. We are using version 0.90
as well and recently our traffic surge and our problem was due to the file
descriptor reach, based on the error in the log, so we increase the file
descriptor for the user, max number of processes per user.

This does not mean it would work for your situation but just saying you
can start to investigate from the log.

Hope it helps.

Jason

On Fri, Oct 24, 2014 at 2:00 AM, Eike Dehling eike@buzzcapture.com
wrote:

Hi ivan,

Thanks for the quick reply.

We are doing a lot of aggregations. And those are often the direct cause
for a crash: a particularly large aggregation using more than available
memory.

Field data is around 10gb, of the 31gb assigned to each node, field data
does not change much after our initial warmers are loaded.

I had expected that the recent addition of two nodes to the cluster would
give us enough resources to run trouble-free. Each node now has less shards
(from 6 nodes with 4 shards, to 8 nodes with 3 shards), and as such
searches and aggregates over one quarter less data? Is this simple math
true, or am i missing something?

How is field data expected to grow or shrink when going from 4 to 3
shards on a node? Does it use about one quarter less memory?

Eike

On Thursday, October 23, 2014, Ivan Brusic ivan@brusic.com wrote:

There is more to the issue than merely your configuration. What are your
queries? Are you doing a lot of aggregations, especially on on
high-cardinality fields. What kind of hardware are you running now?

Using the API, looks at your field cache usage. The field cache is held
within the Java heap space, which would cause the out of memory issues.

Cheers,

Ivan

On Thu, Oct 23, 2014 at 1:26 PM, Eike Dehling eike@buzzcapture.com
wrote:

Hi all,

We are using elasticsearch in our production backing dashboards with
social media data. We are running 0.90 version.

Our volume of data grows every day, so we semi-regularly add servers to
our cluster to keep things running smooth. A year ago we went from 4 nodes
to 6 nodes. One month ago we have gone from 6 to 8 nodes. Only now, we are
seeing OutOfMemory like issues again.

My question: It surprises me that we are hitting resource limits again
so soon, increase in data does not explain that very well. Any suggestions
for causes?

Best regards,
Eike dehling

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b7ca0070-84bc-4325-8a36-f8600289ae65%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/w7aXHMa8Sls/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAnMdwVi-xq3oXEsYKROtcDwhR58pnZFsShLhmXvVg5VA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAnMdwVi-xq3oXEsYKROtcDwhR58pnZFsShLhmXvVg5VA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Met vriendelijke groet,
Kind regards,

Eike Dehling
Lead Developer

Buzzcapture
Herengracht 180, 1016 BR, Amsterdam

T: +31 (0)20 3200377
M: +31 (0)6 45144840

LinkedIn | @buzzcapture

Recent
01-08-2014: Buzzcapture breidt uit! Bekijk de vacatures
https://www.buzzcapture.com/over-ons/werken-bij/op de site
30-07-2014: Tips voor je contentkalender
https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/ van
augustus en leestips
https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/ voor
op het strand
01-07-2014: EY en Tennet starten met Buzzcapture
28-06-2014: Ziekenhuizen LUMC en MUMC aan de slag met Buzzcapture
01-06-2014: Buzzcapture integreert print
https://www.buzzcapture.com/blog/buzzcapture-biedt-en-online-geintegreerde-monitoring-aan/ content
in Social Media Dashboard

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAMp7VWTUk_iVpdFh9yNZWDFv-vz%3DWOW0m3ut-%2B2uXeL3OGgtFQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAMp7VWTUk_iVpdFh9yNZWDFv-vz%3DWOW0m3ut-%2B2uXeL3OGgtFQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/w7aXHMa8Sls/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHO4itw28qqSeDs5tDHQcROXX0TZLF6%2BY4k%3DXjQizRjPs4TR%2Bw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHO4itw28qqSeDs5tDHQcROXX0TZLF6%2BY4k%3DXjQizRjPs4TR%2Bw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Met vriendelijke groet,
Kind regards,

Eike Dehling
Lead Developer

Buzzcapture
Herengracht 180, 1016 BR, Amsterdam

T: +31 (0)20 3200377
M: +31 (0)6 45144840

LinkedIn | @buzzcapture

Recent
01-08-2014: Buzzcapture breidt uit! Bekijk de vacatures
https://www.buzzcapture.com/over-ons/werken-bij/op de site
30-07-2014: Tips voor je contentkalender
https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/
van
augustus en leestips
https://www.buzzcapture.com/2014/07/contentkalender-augustus-2014-leesvoer-voor-op-het-strand/
voor
op het strand
01-07-2014: EY en Tennet starten met Buzzcapture
28-06-2014: Ziekenhuizen LUMC en MUMC aan de slag met Buzzcapture
01-06-2014: Buzzcapture integreert print
https://www.buzzcapture.com/blog/buzzcapture-biedt-en-online-geintegreerde-monitoring-aan/
content
in Social Media Dashboard

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMp7VWR7xtVRutY%3DXkZSbpk41KT8r_NBjKihGU1HGYbd1nK5sQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.