I have a 2 node development cluster in EC2 on c3.2xlarge AMIs. That means
8 vCPUs, 15GB RAM, 1Gb network, and I have 2 500GB EBS volumes for
Elasticsearch data on each AMI.
I'm running Java 1.7.0_55, and using the G1 collector. The Java args are:
/usr/bin/java -Xms8g -Xmx8g -Xss256k -Djava.awt.headless=true -server
-XX:+UseCompressedOops -XX:+UseG1GC -XX:MaxGCPauseMillis=20
The index has 2 shards, each with 1 replica.
I have a daily index being filled with application log data. The index, on
average, gets to be about:
53.1GB (primary size)
106.2GB (total size)
Other than indexing, there really is nothing going on in the cluster. No
searches, or percolators, just collecting data.
The heap sits about 7GB used.
The issue is that nodes run out of memory. It has run out on merges, and
[2014-07-07 04:38:47,900][WARN ][index.engine.internal ]
[elasticsearch-ip-10-0-0-45] [derbysoft-20140704] failed to flush after
setting shard to inactive
[derbysoft-20140704] Flush failed
[2014-07-07 09:16:31,492][WARN ][cluster.action.shard ]
[elasticsearch-ip-10-0-0-45] [derbysoft-20140707] received shard failed
for [derbysoft-20140707], node[PAqdsZSnSvewPWPPZbQTyw], [P], s[STARTED],
indexUUID [E1WJZUtbTQi8fs29cJbfjQ], reason [engine failure, message [merge
exception][MergeException[java.lang.OutOfMemoryError: Java heap space];
nested: OutOfMemoryError[Java heap space]; ]
- changed the cache field type from 'resident' to 'soft'
- change the index refresh_interval from 1s to 60s
- created a default template for the index such that _all is disabled,
and all fields in the mapping are set to "not_analyzed".
Even with that, the previous days index reports 4.4GB of Field Data.
The Index rate is on average about 4K/s, with no peaks higher than 6.75K/s.
I'm wondering if I'm just pushing the limits of my 2 8GB RAM instances with
an index of this size? Is there something else I could tweak to keep the
server from running out of memory?
I appreciate everyone who actually made it to the end of this long email,
but I wanted to make sure I provided all the relevant data I could think
Thank you very much for your time.
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAND3Dph_vqTUQjag6Hey3atw4BDKhX%2B2gLi0zOYv8n2k8_jskQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.