I'm doing some initial POC with es building a single index into elastic
search. Here is my setup running elastic search out of the box while
chanting the memory params.
JVM: Java 7 jre 1.7_76
GC: CMS
OS: Windows 7
OS Mem: 32 GB
ES mem: 10 GB
Index Size: 3.8 GB
of docs in index: 159,316,035
Single node cluster.
5 primary shards are assigned to this cluster each approx (800mb).
After consuming the log file and doing nothing else (ie no queries are
executed) I notice two things in regards to memory usage. 1) The memory
usage spikes up to 10GB and never goes back down. 2) Restarting ES will
trim down the mem usage back down toward 4.5GG (4.2 of that is for old gen).
Using the RESTful api and Marvell, I've been trying to figure out why the
memory usage is this high when I haven't executed any searches yet.
Essentially all that my cluster is doing is creating a single index large
index and just idling. No searches have ever been requested against this
cluster.
My impression is the ElasticSearch is loading the index into memory, which
leads me to believe either I'm doing something wrong or this is expected
behavior. My initial impression was that Elastic search wouldn't by default
attempt to load an index into memory unless a search request against the
index is required but I may be wrong on that.
If anyone know can explain what I'm doing wrong it would be greatly
appreciated.
Reading some more and running vmmap to confirm, looks to be memory mapped
file store which is the default for win 64 systems.
On Saturday, January 24, 2015 at 6:33:31 AM UTC-8, austin lee wrote:
I'm doing some initial POC with es building a single index into elastic
search. Here is my setup running Elasticsearch out of the box while
chanting the memory params.
JVM: Java 7 jre 1.7_76
GC: CMS
OS: Windows 7
OS Mem: 32 GB
ES mem: 10 GB
Index Size: 3.8 GB
of docs in index: 159,316,035
Single node cluster.
5 primary shards are assigned to this cluster each approx (800mb).
After consuming the log file and doing nothing else (ie no queries are
executed) I notice two things in regards to memory usage. 1) The memory
usage spikes up to 10GB and never goes back down. 2) Restarting ES will
trim down the mem usage back down toward 4.5GG (4.2 of that is for old gen).
Using the RESTful api and Marvell, I've been trying to figure out why the
memory usage is this high when I haven't executed any searches yet.
Essentially all that my cluster is doing is creating a single index large
index and just idling. No searches have ever been requested against this
cluster.
My impression is the Elasticsearch is loading the index into memory, which
leads me to believe either I'm doing something wrong or this is expected
behavior. My initial impression was that Elastic search wouldn't by default
attempt to load an index into memory unless a search request against the
index is required but I may be wrong on that.
If anyone know can explain what I'm doing wrong it would be greatly
appreciated.
You are likely observing how java heap works. Use a tool like jstat to
check how much the heap is in use to see real usage. Nutshell: java never
returns memory to the OS. You tell it a min it can use and it allocates
that on startup. You tell it a max and it won't allocate more.
Memory mapping is another beast. That consumes tons of virtual memory which
is virtually free on a 64 bit OS. It leaves the decision about which
portion of the file stays in real memory mostly up to the OS.
Reading some more and running vmmap to confirm, looks to be memory mapped
file store which is the default for win 64 systems.
On Saturday, January 24, 2015 at 6:33:31 AM UTC-8, austin lee wrote:
I'm doing some initial POC with es building a single index into elastic
search. Here is my setup running Elasticsearch out of the box while
chanting the memory params.
JVM: Java 7 jre 1.7_76
GC: CMS
OS: Windows 7
OS Mem: 32 GB
ES mem: 10 GB
Index Size: 3.8 GB
of docs in index: 159,316,035
Single node cluster.
5 primary shards are assigned to this cluster each approx (800mb).
After consuming the log file and doing nothing else (ie no queries are
executed) I notice two things in regards to memory usage. 1) The memory
usage spikes up to 10GB and never goes back down. 2) Restarting ES will
trim down the mem usage back down toward 4.5GG (4.2 of that is for old gen).
Using the RESTful api and Marvell, I've been trying to figure out why the
memory usage is this high when I haven't executed any searches yet.
Essentially all that my cluster is doing is creating a single index large
index and just idling. No searches have ever been requested against this
cluster.
My impression is the Elasticsearch is loading the index into memory,
which leads me to believe either I'm doing something wrong or this is
expected behavior. My initial impression was that Elastic search wouldn't
by default attempt to load an index into memory unless a search request
against the index is required but I may be wrong on that.
If anyone know can explain what I'm doing wrong it would be greatly
appreciated.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.