Elasticsearch memory usage

codebird · March 13, 2014, 11:50am

Hello,

I have been using elasticsearch on a ubuntu server for a year now, and everything was going great. I had an index of 150,000,000 entries of domain names, running small queries on it, just filtering by 1 term no sorting no wildcard nothing. Now we moved servers, I have now a CentOS 6 server, 32GB ram and running elasticserach but now we have 2 indices, of about 150 million entries each 32 shards, still running the same queries on them nothing changed in the queries. But since we went online with the new server, I have to restart elasticsearch every 2 hours before OOM killer kills it.

What's happening is that elasticsearch starts using memory till 50% then it goes back down to about 30% gradually then starts to go up again gradually and never goes back down.

I have tried all the solutions I found on the net, I am a developer not a server admin.

I have these setting in my service wrapper configuration

set.default.ES_HOME=/home/elasticsearch
set.default.ES_HEAP_SIZE=8192
set.default.MAX_OPEN_FILES=65535
set.default.MAX_LOCKED_MEMORY=10240
set.default.CONF_DIR=/home/elasticsearch/conf
set.default.WORK_DIR=/home/elasticsearch/tmp
set.default.DIRECT_SIZE=4g

Java Additional Parameters

wrapper.java.additional.1=-Delasticsearch-service
wrapper.java.additional.2=-Des.path.home=%ES_HOME%
wrapper.java.additional.3=-Xss256k
wrapper.java.additional.4=-XX:+UseParNewGC
wrapper.java.additional.5=-XX:+UseConcMarkSweepGC
wrapper.java.additional.6=-XX:CMSInitiatingOccupancyFraction=75
wrapper.java.additional.7=-XX:+UseCMSInitiatingOccupancyOnly
wrapper.java.additional.8=-XX:+HeapDumpOnOutOfMemoryError
wrapper.java.additional.9=-Djava.awt.headless=true
wrapper.java.additional.10=-XX:MinHeapFreeRatio=40
wrapper.java.additional.11=-XX:MaxHeapFreeRatio=70
wrapper.java.additional.12=-XX:CMSInitiatingOccupancyFraction=75
wrapper.java.additional.13=-XX:+UseCMSInitiatingOccupancyOnly
wrapper.java.additional.15=-XX:MaxDirectMemorySize=4g

Initial Java Heap Size (in MB)

wrapper.java.initmemory=%ES_HEAP_SIZE%

And these in elasticsearch.yml
ES_MIN_MEM: 5g
ES_MAX_MEM: 5g
#index.store.type=mmapfs
index.cache.field.type: soft
index.cache.field.max_size: 10000
index.cache.field.expire: 10m
index.term_index_interval: 256
index.term_index_divisor: 5

java version:
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

Elasticsearch version
"version" : {
"number" : "1.0.0",
"build_hash" : "a46900e9c72c0a623d71b54016357d5f94c8ea32",
"build_timestamp" : "2014-02-12T16:18:34Z",
"build_snapshot" : false,
"lucene_version" : "4.6"
}

Using elastica PHP

I have tried playing with values up and down to try to make it work, but nothing is changing.

Please any help would be highly appreciated.

warkolm · March 14, 2014, 11:00am

Your config is a bit of a mess unfortunately, you're setting different
values for Xms and Xmx in a few places which makes it hard to know what is
applied.
Can you do a ps -ef|grep elasticsearch and post it? That will clarify the
memory settings used as least.

That aside you really want to set 50% of your system ram to ES, if the host
is dedicated to ES.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 13 March 2014 22:50, codebird mallah.hicham@gmail.com wrote:

Hello,

I have been using elasticsearch on a ubuntu server for a year now, and
everything was going great. I had an index of 150,000,000 entries of domain
names, running small queries on it, just filtering by 1 term no sorting no
wildcard nothing. Now we moved servers, I have now a CentOS 6 server, 32GB
ram and running elasticserach but now we have 2 indices, of about 150
million entries each 32 shards, still running the same queries on them
nothing changed in the queries. But since we went online with the new
server, I have to restart elasticsearch every 2 hours before OOM killer
kills it.

What's happening is that elasticsearch starts using memory till 50% then it
goes back down to about 30% gradually then starts to go up again gradually
and never goes back down.

I have tried all the solutions I found on the net, I am a developer not a
server admin.

I have these setting in my service wrapper configuration

set.default.ES_HOME=/home/elasticsearch
set.default.ES_HEAP_SIZE=8192
set.default.MAX_OPEN_FILES=65535
set.default.MAX_LOCKED_MEMORY=10240
set.default.CONF_DIR=/home/elasticsearch/conf
set.default.WORK_DIR=/home/elasticsearch/tmp
set.default.DIRECT_SIZE=4g

Java Additional Parameters

wrapper.java.additional.1=-Delasticsearch-service
wrapper.java.additional.2=-Des.path.home=%ES_HOME%
wrapper.java.additional.3=-Xss256k
wrapper.java.additional.4=-XX:+UseParNewGC
wrapper.java.additional.5=-XX:+UseConcMarkSweepGC
wrapper.java.additional.6=-XX:CMSInitiatingOccupancyFraction=75
wrapper.java.additional.7=-XX:+UseCMSInitiatingOccupancyOnly
wrapper.java.additional.8=-XX:+HeapDumpOnOutOfMemoryError
wrapper.java.additional.9=-Djava.awt.headless=true
wrapper.java.additional.10=-XX:MinHeapFreeRatio=40
wrapper.java.additional.11=-XX:MaxHeapFreeRatio=70
wrapper.java.additional.12=-XX:CMSInitiatingOccupancyFraction=75
wrapper.java.additional.13=-XX:+UseCMSInitiatingOccupancyOnly
wrapper.java.additional.15=-XX:MaxDirectMemorySize=4g

Initial Java Heap Size (in MB)

wrapper.java.initmemory=%ES_HEAP_SIZE%

And these in elasticsearch.yml
ES_MIN_MEM: 5g
ES_MAX_MEM: 5g
#index.store.type=mmapfs
index.cache.field.type: soft
index.cache.field.max_size: 10000
index.cache.field.expire: 10m
index.term_index_interval: 256
index.term_index_divisor: 5

java version:
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

Elasticsearch version
"version" : {
"number" : "1.0.0",
"build_hash" : "a46900e9c72c0a623d71b54016357d5f94c8ea32",
"build_timestamp" : "2014-02-12T16:18:34Z",
"build_snapshot" : false,
"lucene_version" : "4.6"
}

Using elastica PHP

I have tried playing with values up and down to try to make it work, but
nothing is changing.

Please any help would be highly appreciated.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-memory-usage-tp4051793.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1394711400299-4051793.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aqfFB3ivJr08JHFWW6kA1J67SOk4f8eA5EZB8wEfJ5BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

codebird · March 14, 2014, 11:10am

Hello Mark, thanks for the reply

The host isn't dedicated to ES, there is mysql and apache on it, but there is not too much load from their side, so I can set it to 50% of the RAM I just lowered it to see if this will fix my issue.

Here's the output of ps -ef | grep elasticsearch

/home/elasticsearch/bin/service/exec/elasticsearch-linux-x86-64 /home/elasticsearch/bin/service/elasticsearch.conf wrapper.syslog.ident=elasticsearch wrapper.pidfile=/home/elasticsearch/bin/service/./elasticsearch.pid wrapper.name=elasticsearch wrapper.displayname=Elasticsearch wrapper.daemonize=TRUE wrapper.statusfile=/home/elasticsearch/bin/service/./elasticsearch.status wrapper.java.statusfile=/home/elasticsearch/bin/service/./elasticsearch.java.status wrapper.lockfile=/var/lock/subsys/elasticsearch wrapper.script.version=3.5.14
root 23802 23798 99 03:52 ? 07:55:57 /usr/java/jdk1.7.0_25/bin/java -Delasticsearch-service -Des.path.home=/home/elasticsearch -Xss256k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -XX:MinHeapFreeRatio=40 -XX:MaxHeapFreeRatio=70 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:MaxDirectMemorySize=4g -Xms8192m -Xmx8192m -Djava.library.path=/home/elasticsearch/bin/service/lib -classpath /home/elasticsearch/bin/service/lib/wrapper.jar:/home/elasticsearch/lib/elasticsearch-1.0.0.jar:/home/elasticsearch/lib/elasticsearch-1.0.0.jar:/home/elasticsearch/lib/jna-3.3.0.jar:/home/elasticsearch/lib/jts-1.12.jar:/home/elasticsearch/lib/log4j-1.2.17.jar:/home/elasticsearch/lib/lucene-analyzers-common-4.6.1.jar:/home/elasticsearch/lib/lucene-codecs-4.6.1.jar:/home/elasticsearch/lib/lucene-core-4.6.1.jar:/home/elasticsearch/lib/lucene-grouping-4.6.1.jar:/home/elasticsearch/lib/lucene-highlighter-4.6.1.jar:/home/elasticsearch/lib/lucene-join-4.6.1.jar:/home/elasticsearch/lib/lucene-memory-4.6.1.jar:/home/elasticsearch/lib/lucene-misc-4.6.1.jar:/home/elasticsearch/lib/lucene-queries-4.6.1.jar:/home/elasticsearch/lib/lucene-queryparser-4.6.1.jar:/home/elasticsearch/lib/lucene-sandbox-4.6.1.jar:/home/elasticsearch/lib/lucene-spatial-4.6.1.jar:/home/elasticsearch/lib/lucene-suggest-4.6.1.jar:/home/elasticsearch/lib/mysql-connector-java-5.1.26-bin.jar:/home/elasticsearch/lib/spatial4j-0.3.jar:/home/elasticsearch/lib/sigar/sigar-1.6.4.jar -Dwrapper.key=DFYAQvFT98OPKWIu -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=23798 -Dwrapper.version=3.5.14 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.elasticsearch.bootstrap.ElasticsearchF

ciprian · June 11, 2014, 8:33am

Hi,

I'm a beginner in elasticsearch. I'm using also ubuntu for elasticsearch(version 1.1.0). My issue : where should I set the ES_MIN/MAX_SIZE(= ES_HEAP_SIZE)? In /etc/elasticsearch/elasticsearch.yml or /etc/default/elasticsearch or /etc/init.d/elasticsearch? From what i read i should set the ES_MIN/MAX_SIZE = ram / 2.
My application is reading data from a Sqs and pushing it to elasticseach. This is done from 2 consumers each with a thread pool of 5 threads. That means 10 threads who are inserting documents in elasticsearch. I have a separate server for elasticsearch with 8gb of ram.
From my configuration : I set ES_MIN/MAX_SIZE = 4g(=ram / 2) only in /etc/elasticsearch/elasticsearch.yml and restarting the elasticsearch. When I call the htop, I see for elasticsearch user the next command which is duplicated(..10 threads => more line like this one) : /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt. Shouldn't it be -Xms2g -Xmx2g, because I set in elasticsearch.yml 2g?
From what i read in your post, you set in "service wrapper configuration" set.default.ES_HEAP_SIZE=8192.
First of all : which file is linked to "service wrapper configuration" /etc/default/elasticsearch or /etc/init.d/elasticsearch?
Second : why you have different values in elasticsearch.yml ES_MIN_MEM: 5g ES_MAX_MEM: 5g than in "service wrapper configuration" (ES_HEAP_SIZE=8192) ?
Third : where should i set the wrapper.java.additional.1 ... 15 and wrapper.java.initmemory=%ES_HEAP_SIZE% ?