Before you take off for your well-deserved holidays, here are some tips on how to avoid being rudely pulled back by a pager alert.
Make sure you set minimum_master_nodes
To prevent a split brain, check you have set
discovery.zen.minimum_master_nodes in your
elasticsearch.yml. In a typical cluster with three master-eligible (or dedicated master) nodes, this setting should be set to 2. See the
Check your JVM heap size
Elasticsearch is using two types of memory: The JVM heap, the size of which is set explicitly, and the file system cache, which is set implicitly (all the memory not assigned to the JVM). What memory you assign to the JVM cannot be used by the cache, so it should be as low as possible, and definitely not above 30.5 GB. Read the classic "A Heap of Trouble" blog post on this topic to learn more.
Keep your shards in line
A cluster is only as healthy as its shards. In general, you want to have as few shards as you can get away with. Well-sized shards are usually at least a few gigabytes large. To control your shard size, look to the
number_of_shards index setting, as well as the total number of indices in your cluster.
And lastly, if you're not using it yet, you should: X-Pack Monitoring is free and collects invaluable information about your cluster. Head over to the download page to install X-Pack, and get your free license here if you don't have one already.
And with that, I wish you a happy (and hopefully alert-free) holiday season!