I have the following situation. I have 3 nodes cluster, each node has 32 GB
DDR3 RAM and 2 Octocore Processors (16 cores), with HT. Allocated 50% of
the RAM to Elasticsearch. We have only 1 index with 32 shards and 1 replica
each, making it 64 shards in the cluster. We have a lot of mappings in the
index (more than 3000 ~ no. of users).
We have a lot of updates queries. In fact, we have only update queries with
upserts. We are using routing to put similar data in the same shard. We see
the rate of update queries ranging between 20/sec to 60/sec. This is going
to increase to about 130-140/sec when we go live. Our queries are mostly
filtered queries with a lot of use of term faceting (now we are using the
We are using doc_values as much as possible to reduce bringing of field
data in the field data cache.
For the last 2 weeks, I have been noticing that the memory usage would
never drop. Every node in the cluster is using about 90% of the memory all
the time. This triggers the GC a lot (sometimes more than 5 times in a
minute). We are using G1GC instead of the default CMS.
I then decided to stop querying and indexing both. The memory usage still
does not drop and there are still a lot of GCs in a given minute. The
effect of this is visible in the queue we are using to update documents. I
can clearly relate the spikes in the queue with the GC activity i.e.
whenever GC happens, indexing seems to become slow. This is sort of
I am attaching screenshots of BigDesk of two of our nodes in the cluster (1
master data node and 1 non-master data node).
What could be the reason? How should I debug this issue? Will be happy to
share more details if needed.
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firstname.lastname@example.org.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACWtv5k%3Dy9%2BY93jFQUV%2Bp5TM9D-rS-2HF1cOqPPTEatFrwjMPQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.