Understanding Heap Usage for Indexing and Updating


Following is the document structure of the document I am indexing in ES.

"key1": "val1",
"key2": "val2",
"keyN": "valN",
"array1": [
"array2": [
"key1": "val1",
"key2": "val2",
"key3": "val3",
"key4": "val4"

Document size usually is about 800 bytes but it can be between 600 to 1500
bytes depending upon the number of entries of we have in the arrays (that
can be anywhere between 1 to 20) and there is a URL field whose value can
be small or large depending upon user input.

The way I am using elasticsearch is that I do a large number of writes,
1000 - 1300 per second. But the way I do it is using UPDATE API with UPSERT
instead of directly indexing. The reason for this is that sometimes I get
documents that have already been indexed. So I generate the document ID
(DOC_ID) by doing a md5 of some unique values in my doc and always do an
UPDATE with UPSERT so that the doc gets index if it does not exist, but if
it does exist it must get dropped.

Out of these 1000 - 1300 UPDATES per second, about 80% UPDATES are of the
above type as I have described.

In the remaining 20%, I do updates on the docs that have been already been
indexed (by the above described way of indexing docs). How this is done is
by using an UPDATE script like so:

POST /index/doc_type/DOC_ID/_update
"script": "if (ctx._source.array1.contains(some_val)) { ctx.op =
"none" } else { ctx._source.array1 += some_val; ctx._source._array2 +=
some_doc }",
"params": {
"some_val": ,
"some_doc": {
"key1": "val1",
"key2": "val2",

So if a document DOC_ID has been indexed, then in the second type I just
update that document otherwise I do nothing. Also, if the document is
found, then I do some more checks to perform the operation as described by
the script in the above UPDATE request.

I am not doing a large number of queries right now since the application is
just getting developed but that could go upto 100 - 200 requests per
second. Query types are count (i.e. we don't need the docs but just the
numbers) with a lot of term facets.

This is pretty much my scenario.

Now coming to the problem, the issues that I am seeing is that the write
throughput was high when I started the cluster but with time it slowed down
to about 50 writes (UPDATES in my case) per second.

There is just one index with 32 shards and 1 replica each. I have a 3 node
cluster with 32 GB of RAM, out of which I have committed 16 GB to JVM heap
(MLOCKALL set to true as suggested to prevent paging out). Each node has 32
cores. We are using normal SATA II hard disks and not SSDs. I have also
installed ElasticSearchHQ plugin for monitoring.

Following are the index settings:

"index": {
"settings": {
"index.refresh_interval": "5s",
"index.version.created": "900799",
"index.number_of_replicas": "1",
"index.uuid": "4dFGm17qSF6Qe8lc20rgbw",
"index.number_of_shards": "32"

Some observations:

  1. After sometime, 96% of JVM heap is always used by ES.
  2. GC has been taking a lot of time (5 sec to 18 sec)
  3. After GC memory drop is negligible. (Example 15.7 GB to 15.6 GB)
  4. Index Refresh time is large (in the order of 100s of milliseconds)

The above observations did not changed when I stopped writing and querying
to ES for more than 12 hours (I stopped the writer at night and checked HQ
the next morning).

I am struggling hard to understand how ES is using the JVM that the heap
usage does not drop at all even when nothing is being done. Why is GC so
slow? The node configuration is pretty decent and I think I have configured
ES to at least all the suggested best configs (like mlockall set to true,
appropriate heap size and enough memory for disk cache, ulimit set to

What else should I be looking at? Or is there more information for you guys
to help out?

Some help will be greatly appreciated!

Vaidik Kapoor

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACWtv5mOfP2AwjLqQ6m%2BMbn_JG_x7A0LPbswrGwFEr-77MhyZg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Im facing the exactly same problem, any solution to this?

(Please correct me if I'm wrong) doing lots of updates is really expensive in terms of memory use because Elasticsearch retrieves the current version of the document and keeps this in memory until the index is flushed. Therefore multiple updates to the same document will cause multiple copies to be held in memory.

Is it definitely not possible to use the /_bulk end point with the "index" operation to re-index the documents instead?

thanks for your help!

First of all, we don't do multiple updates to the same document, because each of our batch does one operation to each document _id. Although in the next batch we might update that same _id (10min later).

About the /_bulk: could you explain better what you mean?
We use the /_bulk (via python) but doing the "update" operation, because we use scripts. How would this "index" to re-index help in such case?

Using the index operation wouldn't be appropriate if you're using scripts to update particular portions of a document.

I tend to use the "index" operation because Elasticsearch isn't the primary store and all the updates come in the form of a complete document.

I've also noticed that once an Elasticsearch node is using 80% or so of its heap it gets blocked doing garbage collection and doesn't tend to recover. Unfortunately, I'm not aware of a fix once this occurs other than rebooting the node perhaps but that could lose data or cause searches to fail.

Reducing the number of replica shards resolved the issue (for now) but that won't help in your case as there is only one replica per shard.

Another point is that even doing a flush "POST http://ip:9200/_all/_flush" with { force: true } the JVM memory does not go down. And I stopped all the process that were inserting in ES. I really don't know how to proceed here.

I've also tried adjusting the index settings of particular fields to reduce overheads e.g.

include_in_all: false | as these field values won't be searched
index_options: "docs" | as these field values won't be used in scoring or highlighting
norms: {"enabled": false} | as these field values won't be used in scoring
store: true | as these field values should be individually retrievable i.e. separately from parsing _source
term_vector: "no" | as these field values should not be analyzed

Filipe, how many replica shards do you have? my previous comment referred to the original post. you may be able to reduce the number of replicas to resolve the memory issue.

How much ram does each node have and what is the ES_HEAP_SIZE set to?

We have 1 replica, 3 nodes, ES_HEAP_SIZE = 7gb (50% of the total). I'm thinking that our cluster cannot free this memory itself, I will try to restart the node.

Each node is on a m4.xlarge on EC2

Edit 1: Here's the template that we are using https://s3.amazonaws.com/hkn-temp/template.txt

Edit 2: Here's what the HQ shows me in one of the nodes:

Heap Used: 6.6GB
Heap Committed: 7.0GB
Non Heap Used: 821.1MB
Non Heap Committed: 1.2GB
JVM Uptime: 02:51:43
Thread Count/Peak: 63 / 67
GC (Young) Count: 4319
GC (Young)Time: 00:02:50
GC (Old) Count: 3005
GC (Old)Time: 00:19:51
Java Version: 1.8.0_91
JVM Vendor: Oracle Corporation
JVM: OpenJDK 64-Bit Server VM

Edit 4: Restarting our cluster free up all the jvm memory. Not a good solution tough.