We have a cluster that has three indexes (one main, and two supporting) with 5 shards and one replica. I am using 7 Windows nodes, each of which have 8 cores and 56gb RAM. I have set the heap to use 28gb. I have disabled the paging file in the OS and set mlock to true. No matter what I have tried, I am still seeing swap being used. ElasticHQ is reporting between 19 - 32 mb swap being used on the nodes.
We have found that we have far exceeded the recommended shard size of 30 - 50gb. Our shards are currently 70 - 90 gb. We are gearing up to reindex and we will split out to more shards at that time. The swap is an issue we have had since we started, however (i.e. even when the shards will smaller).
We are seeing some perf issues, and I can't help but be concerned about this metric that ElasticHQ is flagging in red. I would like to try to resolve this, but haven't had any luck yet.