Hey everyone, while I'm extremely familiar with Elasticsearch and use it daily, I was hoping some of you that are more familiar with JVM tuning could help me out.
I setup clusters constantly (on Ubuntu 12.04 if it matters), and find that using the default value for CMSInitiatingOccupancyFraction (75% I think) in my elasticsearch init file usually works just fine. However, there's a particular cluster I've just set up that has a super high write/read ratio. ~1k logs are dumped per minute across two m1.smalls and rarely searched (maybe a few times an hour max). The hardware selection is obviously not optimal, but budget restricts moving to a bigger box size or adding more nodes.
With this cluster, I've ran into several OutOfMemory scenarios that require a restart of the process. Heap used is perpetually around 90%. I'm aware that there's generally nothing you can do to alleviate OutOfMemory exceptions when you're using such low-end hardware, but is there any chance that lowering CMSInitiatingOccupancyFraction to around 30-50 would help?
The GC MarkSweep duration is already 8-12s, so I guess I'm just worried that increasing the number/duration of GC cycles will result in poorer performance. But, considering how often hard restarts of the cluster are required now, a slight performance loss in favor of keeping the cluster alive is preferred.
Thanks in advanced.