Elasticsearch memory usage

Elastic Stack Elasticsearch
1

Hello,

I have been using elasticsearch on a ubuntu server for a year now, and everything was going great. I had an index of 150,000,000 entries of domain names, running small queries on it, just filtering by 1 term no sorting no wildcard nothing. Now we moved servers, I have now a CentOS 6 server, 32GB ram and running elasticserach but now we have 2 indices, of about 150 million entries each 32 shards, still running the same queries on them nothing changed in the queries. But since we went online with the new server, I have to restart elasticsearch every 2 hours before OOM killer kills it.

What's happening is that elasticsearch starts using memory till 50% then it goes back down to about 30% gradually then starts to go up again gradually and never goes back down.

I have tried all the solutions I found on the net, I am a developer not a server admin.

I have these setting in my service wrapper configuration

set.default.ES_HOME=/home/elasticsearch
set.default.ES_HEAP_SIZE=8192
set.default.MAX_OPEN_FILES=65535
set.default.MAX_LOCKED_MEMORY=10240
set.default.CONF_DIR=/home/elasticsearch/conf
set.default.WORK_DIR=/home/elasticsearch/tmp
set.default.DIRECT_SIZE=4g

Java Additional Parameters

wrapper.java.additional.1=-Delasticsearch-service
wrapper.java.additional.2=-Des.path.home=%ES_HOME%
wrapper.java.additional.3=-Xss256k
wrapper.java.additional.4=-XX:+UseParNewGC
wrapper.java.additional.5=-XX:+UseConcMarkSweepGC
wrapper.java.additional.6=-XX:CMSInitiatingOccupancyFraction=75
wrapper.java.additional.7=-XX:+UseCMSInitiatingOccupancyOnly
wrapper.java.additional.8=-XX:+HeapDumpOnOutOfMemoryError
wrapper.java.additional.9=-Djava.awt.headless=true
wrapper.java.additional.10=-XX:MinHeapFreeRatio=40
wrapper.java.additional.11=-XX:MaxHeapFreeRatio=70
wrapper.java.additional.12=-XX:CMSInitiatingOccupancyFraction=75
wrapper.java.additional.13=-XX:+UseCMSInitiatingOccupancyOnly
wrapper.java.additional.15=-XX:MaxDirectMemorySize=4g

Initial Java Heap Size (in MB)

wrapper.java.initmemory=%ES_HEAP_SIZE%

And these in elasticsearch.yml
ES_MIN_MEM: 5g
ES_MAX_MEM: 5g
#index.store.type=mmapfs
index.cache.field.type: soft
index.cache.field.max_size: 10000
index.cache.field.expire: 10m
index.term_index_interval: 256
index.term_index_divisor: 5

java version:
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

Elasticsearch version
"version" : {
"number" : "1.0.0",
"build_hash" : "a46900e9c72c0a623d71b54016357d5f94c8ea32",
"build_timestamp" : "2014-02-12T16:18:34Z",
"build_snapshot" : false,
"lucene_version" : "4.6"
}

Using elastica PHP

I have tried playing with values up and down to try to make it work, but nothing is changing.

Please any help would be highly appreciated.

2

Your config is a bit of a mess unfortunately, you're setting different
values for Xms and Xmx in a few places which makes it hard to know what is
applied.
Can you do a ps -ef|grep elasticsearch and post it? That will clarify the
memory settings used as least.

That aside you really want to set 50% of your system ram to ES, if the host
is dedicated to ES.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 13 March 2014 22:50, codebird mallah.hicham@gmail.com wrote:

Hello,

I have been using elasticsearch on a ubuntu server for a year now, and
everything was going great. I had an index of 150,000,000 entries of domain
names, running small queries on it, just filtering by 1 term no sorting no
wildcard nothing. Now we moved servers, I have now a CentOS 6 server, 32GB
ram and running elasticserach but now we have 2 indices, of about 150
million entries each 32 shards, still running the same queries on them
nothing changed in the queries. But since we went online with the new
server, I have to restart elasticsearch every 2 hours before OOM killer
kills it.

What's happening is that elasticsearch starts using memory till 50% then it
goes back down to about 30% gradually then starts to go up again gradually
and never goes back down.

I have tried all the solutions I found on the net, I am a developer not a
server admin.

I have these setting in my service wrapper configuration

set.default.ES_HOME=/home/elasticsearch
set.default.ES_HEAP_SIZE=8192
set.default.MAX_OPEN_FILES=65535
set.default.MAX_LOCKED_MEMORY=10240
set.default.CONF_DIR=/home/elasticsearch/conf
set.default.WORK_DIR=/home/elasticsearch/tmp
set.default.DIRECT_SIZE=4g

Java Additional Parameters

wrapper.java.additional.1=-Delasticsearch-service
wrapper.java.additional.2=-Des.path.home=%ES_HOME%
wrapper.java.additional.3=-Xss256k
wrapper.java.additional.4=-XX:+UseParNewGC
wrapper.java.additional.5=-XX:+UseConcMarkSweepGC
wrapper.java.additional.6=-XX:CMSInitiatingOccupancyFraction=75
wrapper.java.additional.7=-XX:+UseCMSInitiatingOccupancyOnly
wrapper.java.additional.8=-XX:+HeapDumpOnOutOfMemoryError
wrapper.java.additional.9=-Djava.awt.headless=true
wrapper.java.additional.10=-XX:MinHeapFreeRatio=40
wrapper.java.additional.11=-XX:MaxHeapFreeRatio=70
wrapper.java.additional.12=-XX:CMSInitiatingOccupancyFraction=75
wrapper.java.additional.13=-XX:+UseCMSInitiatingOccupancyOnly
wrapper.java.additional.15=-XX:MaxDirectMemorySize=4g

Initial Java Heap Size (in MB)

wrapper.java.initmemory=%ES_HEAP_SIZE%

And these in elasticsearch.yml
ES_MIN_MEM: 5g
ES_MAX_MEM: 5g
#index.store.type=mmapfs
index.cache.field.type: soft
index.cache.field.max_size: 10000
index.cache.field.expire: 10m
index.term_index_interval: 256
index.term_index_divisor: 5

java version:
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

Elasticsearch version
"version" : {
"number" : "1.0.0",
"build_hash" : "a46900e9c72c0a623d71b54016357d5f94c8ea32",
"build_timestamp" : "2014-02-12T16:18:34Z",
"build_snapshot" : false,
"lucene_version" : "4.6"
}

Using elastica PHP

I have tried playing with values up and down to try to make it work, but
nothing is changing.

Please any help would be highly appreciated.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-memory-usage-tp4051793.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1394711400299-4051793.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aqfFB3ivJr08JHFWW6kA1J67SOk4f8eA5EZB8wEfJ5BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hello Mark, thanks for the reply

The host isn't dedicated to ES, there is mysql and apache on it, but there is not too much load from their side, so I can set it to 50% of the RAM I just lowered it to see if this will fix my issue.

Here's the output of ps -ef | grep elasticsearch

/home/elasticsearch/bin/service/exec/elasticsearch-linux-x86-64 /home/elasticsearch/bin/service/elasticsearch.conf wrapper.syslog.ident=elasticsearch wrapper.pidfile=/home/elasticsearch/bin/service/./elasticsearch.pid wrapper.name=elasticsearch wrapper.displayname=Elasticsearch wrapper.daemonize=TRUE wrapper.statusfile=/home/elasticsearch/bin/service/./elasticsearch.status wrapper.java.statusfile=/home/elasticsearch/bin/service/./elasticsearch.java.status wrapper.lockfile=/var/lock/subsys/elasticsearch wrapper.script.version=3.5.14
root 23802 23798 99 03:52 ? 07:55:57 /usr/java/jdk1.7.0_25/bin/java -Delasticsearch-service -Des.path.home=/home/elasticsearch -Xss256k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -XX:MinHeapFreeRatio=40 -XX:MaxHeapFreeRatio=70 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:MaxDirectMemorySize=4g -Xms8192m -Xmx8192m -Djava.library.path=/home/elasticsearch/bin/service/lib -classpath /home/elasticsearch/bin/service/lib/wrapper.jar:/home/elasticsearch/lib/elasticsearch-1.0.0.jar:/home/elasticsearch/lib/elasticsearch-1.0.0.jar:/home/elasticsearch/lib/jna-3.3.0.jar:/home/elasticsearch/lib/jts-1.12.jar:/home/elasticsearch/lib/log4j-1.2.17.jar:/home/elasticsearch/lib/lucene-analyzers-common-4.6.1.jar:/home/elasticsearch/lib/lucene-codecs-4.6.1.jar:/home/elasticsearch/lib/lucene-core-4.6.1.jar:/home/elasticsearch/lib/lucene-grouping-4.6.1.jar:/home/elasticsearch/lib/lucene-highlighter-4.6.1.jar:/home/elasticsearch/lib/lucene-join-4.6.1.jar:/home/elasticsearch/lib/lucene-memory-4.6.1.jar:/home/elasticsearch/lib/lucene-misc-4.6.1.jar:/home/elasticsearch/lib/lucene-queries-4.6.1.jar:/home/elasticsearch/lib/lucene-queryparser-4.6.1.jar:/home/elasticsearch/lib/lucene-sandbox-4.6.1.jar:/home/elasticsearch/lib/lucene-spatial-4.6.1.jar:/home/elasticsearch/lib/lucene-suggest-4.6.1.jar:/home/elasticsearch/lib/mysql-connector-java-5.1.26-bin.jar:/home/elasticsearch/lib/spatial4j-0.3.jar:/home/elasticsearch/lib/sigar/sigar-1.6.4.jar -Dwrapper.key=DFYAQvFT98OPKWIu -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=23798 -Dwrapper.version=3.5.14 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.elasticsearch.bootstrap.ElasticsearchF

4

Hi,

I'm a beginner in elasticsearch. I'm using also ubuntu for elasticsearch(version 1.1.0). My issue : where should I set the ES_MIN/MAX_SIZE(= ES_HEAP_SIZE)? In /etc/elasticsearch/elasticsearch.yml or /etc/default/elasticsearch or /etc/init.d/elasticsearch? From what i read i should set the ES_MIN/MAX_SIZE = ram / 2.
My application is reading data from a Sqs and pushing it to elasticseach. This is done from 2 consumers each with a thread pool of 5 threads. That means 10 threads who are inserting documents in elasticsearch. I have a separate server for elasticsearch with 8gb of ram.
From my configuration : I set ES_MIN/MAX_SIZE = 4g(=ram / 2) only in /etc/elasticsearch/elasticsearch.yml and restarting the elasticsearch. When I call the htop, I see for elasticsearch user the next command which is duplicated(..10 threads => more line like this one) : /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt. Shouldn't it be -Xms2g -Xmx2g, because I set in elasticsearch.yml 2g?
From what i read in your post, you set in "service wrapper configuration" set.default.ES_HEAP_SIZE=8192.
First of all : which file is linked to "service wrapper configuration" /etc/default/elasticsearch or /etc/init.d/elasticsearch?
Second : why you have different values in elasticsearch.yml ES_MIN_MEM: 5g ES_MAX_MEM: 5g than in "service wrapper configuration" (ES_HEAP_SIZE=8192) ?
Third : where should i set the wrapper.java.additional.1 ... 15 and wrapper.java.initmemory=%ES_HEAP_SIZE% ?