I'm using Hot-Warm Elasticsearch to store and perform search on logs for my company.
My ES cluster has 8 hot nodes and 10 warm nodes located on 10 servers.
Each server has 64GB RAM.
2 servers that only run warm nodes work fine. I use the recommended setting on these two servers that gives 32GB RAM to ES heap and let the remain free.
The other 8 servers have both SSD and HDD disk so I run 1 hot node and 1 warm node.
I give 20GB RAM to hot node and 24GB RAM to warm node and let just 20GB RAM free.
Everyday, ES creates 12 indices, each for every 2 hours. Each index has 8 shards located on 8 hot nodes.
My daily curator settings are:
- Move all indices older than 2 days to warm.
- Close all indices older than 30 days.
- Delete all indices older than 50 days.
Each indices' size is about 250-320GB.
All open indices' size for 30 days is 100TB.
Here are some problems that I encountered:
- Slow search perform.
- Heap on warm nodes is always red (>90% used)
I have some questions:
- How can I make heap on warm nodes healthy? (<90% used)
- Add more RAM? But it is recommended that each machine should have only 64GB RAM. If yes, how much RAM should I add?
- Forcemerge segments? I notice that when a node has ~10000 segments, it uses 15-16GB memory. But when I run forcemerge to max 1 segment/shard on all cluster, each node downs to 2000 segments and only uses 13-14GB memory. But I also found the maximum size of segments in es5.0/lucene6 is 5GB. If I run forcemerge to 1 segment/shard, each segment size will be 30-40GB. So should I run forcemerge to one segment? And is it true that fewer segments will use less memory?
- How can I improve search performance? Are there any tricks on doc_type, mapping, segments... to improve search performance?
Thanks in advance.