How to decrease memory footprint of ElasticSearch

Elastic Stack Elasticsearch
1

Is it me or does ElasticSearch require a hefty memory footprint even at its
bare minimum? Currently, I have it at

-Xms32 -Xmx32

and it shows that it's using 380m RES. I tried changing it to 16 but it
wouldn't even run. I'm trying to use it for a small project of mine on a
low-memory VPS. Can anyone recommend how to decrease the memory to around
something reasonable like 100m.

--

2

Those resident bits might be the memory-mapped indexes. Try configuring
ElasticSearch to use niofshttp://www.elasticsearch.org/guide/reference/index-modules/store.htmlexplicitly. Use soft
or weak cacheshttp://www.elasticsearch.org/guide/reference/index-modules/cache.html.
Install bigdesk pluginhttp://www.elasticsearch.org/guide/reference/modules/plugins.html,
maybe it will help with figuring out the memory usage. Keep in mind that
JVM needs a lot of RAM for the JIT compiler and the compiled machine code,
try running ElasticSearch with -Xinthttp://www.herongyang.com/JVM/Micro-Benchmark-Interpreted-Only-Mode.html
.

--

3

Hi,

the total process space size can't be limited by -Xms or -Xmx, as they
control only the heap space.

There is no way to prevent a JVM to stay within a process limit, as it must
load objects into the process space. With NIO, the NIO buffers outside of
the heap are accounted for process space.

To get the smallest footprint, try the following

  • disable all plugins
  • create a $HOME/.elasticsearch.in.sh file with your custom JVM params
  • use -d32 for 32-bit JVM (disabling niofs and using simplefs is a natural
    consequence of 32bit)
  • use -Xms64m -Xmx64m -Xss192k (128k for JDK 6), that is the lowest JVM
    start parameters I know that are safe
  • disable all other extra JVM settings in elasticsearch.in.sh (comment them
    out)
  • tune settings in config/elasticsearch.yml: set shards = 1, set replica =
    0, set all thread pools to max size 1, set segment tuning to lowest values
    (you have to experiment, around 1m for segment size should work)
  • tune your API usage, do not use caches for facets or for filters, do not
    use warming
  • check memory footprint with the cluster state API

There is a price you pay, performance will be low and the maximum data
volume too, and the risk of OOM is high.

There are many JVMs out there beside the standard JVM of Oracle which are
targeted at small embedded systems. But, unfortunately, without hacking the
ES source (and the dependencies), it won't be possible to get ES running on
other JVMs.

Jörg

On Monday, November 5, 2012 12:51:24 AM UTC+1, Artem Grinblat wrote:

Those resident bits might be the memory-mapped indexes. Try configuring
Elasticsearch to use niofshttp://www.elasticsearch.org/guide/reference/index-modules/store.htmlexplicitly. Use soft
or weak cacheshttp://www.elasticsearch.org/guide/reference/index-modules/cache.html.
Install bigdesk pluginhttp://www.elasticsearch.org/guide/reference/modules/plugins.html,
maybe it will help with figuring out the memory usage. Keep in mind that
JVM needs a lot of RAM for the JIT compiler and the compiled machine code,
try running Elasticsearch with -Xinthttp://www.herongyang.com/JVM/Micro-Benchmark-Interpreted-Only-Mode.html
.

--

1 Like