Elasticsearch Memory issue


(AlexK) #1

Hi,

I am using ES version 1.1.1 on a single node with the below settings:

  • 2 shards per indices;
  • currently 395 indices;
  • currently 428GB;
  • 18GB Heap commited on a 24 GB machine;

I used it only for indexing now, because I noted some memory issues when
performing search: elasticsearch became unresponsive...
My requirement curently is to use ElasticSearch only on a single machine
configuration.

I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge Collector
very often) whithout success.

Does anyone have some advices for me ?

  1. Is the only way is to increase the heap (do you have a ratio RAM / data
    size) ?
  2. Does adding additional nodes on the same machine will help, why ?
  3. Final question, I cannot see why in the indexing part so much of the RAM
    is used, if anybody can explain ?

Thanks a lot for your answers.
Alex.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

You are more than likely reaching the limits of the node.
Your options are to delete data, add more RAM (you should have 50% of
system for heap), close some old indexes or add nodes. Adding more nodes
spreads the shards of your indexes across the cluster which is essentially
spreading the load.

You could try disabling bloom filter cache as this will reduce your memory
usage a little, take a look Elasticsearch Curator which can do this for you.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 1 July 2014 01:04, AlexK alexandre.kapell@gmail.com wrote:

Hi,

I am using ES version 1.1.1 on a single node with the below settings:

  • 2 shards per indices;
  • currently 395 indices;
  • currently 428GB;
  • 18GB Heap commited on a 24 GB machine;

I used it only for indexing now, because I noted some memory issues when
performing search: elasticsearch became unresponsive...
My requirement curently is to use ElasticSearch only on a single machine
configuration.

I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge
Collector very often) whithout success.

Does anyone have some advices for me ?

  1. Is the only way is to increase the heap (do you have a ratio RAM / data
    size) ?
  2. Does adding additional nodes on the same machine will help, why ?
  3. Final question, I cannot see why in the indexing part so much of the
    RAM is used, if anybody can explain ?

Thanks a lot for your answers.
Alex.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YSdjUQnQJ_Wwu82iHxT5m%3DZVDRR_pskCXQ2RPXmLHRxQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(AlexK) #3

Hi Mark,

Thanks for your quick answer.
I cannot increase the RAM for ES, as I am already using 75% of the ram for
the JVM.
I will take a look at disabling the bloom filter cache to see if that
change anything.

Regarding the option of adding more nodes:

  • Do you have an idea of how many nodes are required to sustain that
    quantity of data (428Gb), I will say 3 nodes one for each indice and an
    extra node for replica, do you agree ?
  • In the hypothesis that I will not use in a first time replicas, do you
    have an idea about the quantity of RAM needed for the two nodes (continuous
    indexing and few search requests) ?

Again, thanks for your answers.
Alex.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #4

When it comes to capacity the answer is, it depends.
Given you're at around 430GB on a single node now, I'd add another node and
then see how things look at around the 8-900GB mark (spread across both).

Another clarification; The recommended operating procedure is to use half
your system RAM for heap, leaving the other half for OS caching, which
increases performance of ES. In your case under the best possible
circumstances, you should only really be using 12GB of your total 24GB.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 1 July 2014 17:30, AlexK alexandre.kapell@gmail.com wrote:

Hi Mark,

Thanks for your quick answer.
I cannot increase the RAM for ES, as I am already using 75% of the ram for
the JVM.
I will take a look at disabling the bloom filter cache to see if that
change anything.

Regarding the option of adding more nodes:

  • Do you have an idea of how many nodes are required to sustain that
    quantity of data (428Gb), I will say 3 nodes one for each indice and an
    extra node for replica, do you agree ?
  • In the hypothesis that I will not use in a first time replicas, do you
    have an idea about the quantity of RAM needed for the two nodes (continuous
    indexing and few search requests) ?

Again, thanks for your answers.
Alex.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Z5dt7GP%3DnhfijLBUc_nuaswMQ6gzGgG0FuM4i4BfSMoA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jilles van Gurp) #5

You should tweak cache sizes. At least the field data cache needs to be
restricted (unbounded by default). Also, ensuring the various circuit
breakers are turned on will help. Another tip is to disable the _all field
if you don't need it.

All this should reduce the amount of memory ES uses and make it less likely
your cluster becomes unavailable. We use Elasticsearch with Kibana in our
production setup. Things definitely do not fail gracefully if you run short
of memory; so you need to prevent that situation. I've had a completely
unresponsive cluster on two occasions. With the current settings, it has
been running stable for several weeks now.

I've learned a few of these things the hard way. I think a ES tuning guide
for non experts is desperately needed. The out of the box experience is not
really appropriate for any serious production environment. But then, you
wouldn't run mysql with default settings in production either. In my
experience, you currently need to piece together bits of good advice spread
through the documentation and various forum posts. If you have an untuned
Elasticsearch in production there are several failure scenarios that are
likely to result in unavailability and data loss. Especially if you are
using ELK with lots of log data, you need to tune or you basically will
have a dead cluster in no time due to OOMs.

Jilles

On Monday, June 30, 2014 5:04:12 PM UTC+2, AlexK wrote:

Hi,

I am using ES version 1.1.1 on a single node with the below settings:

  • 2 shards per indices;
  • currently 395 indices;
  • currently 428GB;
  • 18GB Heap commited on a 24 GB machine;

I used it only for indexing now, because I noted some memory issues when
performing search: elasticsearch became unresponsive...
My requirement curently is to use ElasticSearch only on a single machine
configuration.

I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge
Collector very often) whithout success.

Does anyone have some advices for me ?

  1. Is the only way is to increase the heap (do you have a ratio RAM / data
    size) ?
  2. Does adding additional nodes on the same machine will help, why ?
  3. Final question, I cannot see why in the indexing part so much of the
    RAM is used, if anybody can explain ?

Thanks a lot for your answers.
Alex.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5e79bff1-409e-4389-8be3-69590f3ffb56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(AlexK) #6

Thanks both of you for the answer.

I have just decrease the Java Jvm memory to 50% (12G), I will see if that
helps.

@Jilles:

  • I am using the defaut Logstash template and I things that by default the
    _all field is disable...Ah no that not the case :frowning: I will correct this
    settings but why by default this field is enabled ?
  • I will set a limit for the field cache. But regarding the circuit
    breaker, what are these ?
  • I totally agree that it would be good to have a deep tuning guide on ES :slight_smile:

Thanks again,
Alex

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3894e4b5-eb8a-4c0d-90ff-75eb884743f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(smonasco-2) #7

My understanding was that ES 1.1 was using memory mapped files and so the field cache would not be part of the heap.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/308f68ba-bd7a-4775-ae1d-90088445d103%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(AlexK) #8

Setting up the JVM memory to 50% (12G) did not ease the problem as I
noticed GC collection up to 3min :slight_smile:
Will really need to add a bunch of RAM to my machine..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/874f127a-639a-4416-bdec-d50d77920f45%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #9

With ES on a single machine, "tuning" does not cure the symptoms in the
long run. ES was designed to scale out on many nodes, so the simplest path
is to add nodes.

In a restricted environment, you could try to disable features that consume
a fairly amount of resources: disable _source and _all field, bloom codec
filter cache, reduce complex analyzers in the mapping, reduce number of
shards, reduce filter caches ...

Jörg

On Mon, Jun 30, 2014 at 5:04 PM, AlexK alexandre.kapell@gmail.com wrote:

Hi,

I am using ES version 1.1.1 on a single node with the below settings:

  • 2 shards per indices;
  • currently 395 indices;
  • currently 428GB;
  • 18GB Heap commited on a 24 GB machine;

I used it only for indexing now, because I noted some memory issues when
performing search: elasticsearch became unresponsive...
My requirement curently is to use ElasticSearch only on a single machine
configuration.

I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge
Collector very often) whithout success.

Does anyone have some advices for me ?

  1. Is the only way is to increase the heap (do you have a ratio RAM / data
    size) ?
  2. Does adding additional nodes on the same machine will help, why ?
  3. Final question, I cannot see why in the indexing part so much of the
    RAM is used, if anybody can explain ?

Thanks a lot for your answers.
Alex.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHB_jCLt7%3DR4H4O_wy-sAhGx90TrpW%3Du8AHXROdemNyYA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(AlexK) #10

Hi Jorg,

Thanks for the answer.

The idea behind the restriction of a single machine was for instance to
install ELK on a machine and perform fast indexing and review of a set of
log. What I got wrong is that the log size can be important (hundreds of
Gb) so this architecture will not work, according to the answers
above...(goal was to replace Splunk in a similar set up)

Thanks,
Alex.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/94ccef51-1cd9-4210-9f6c-d622108633df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #11