We are using Elasticsearch 2.4 to index and search web pages, and recently it started to get slow.
Currently we have 5 nodes (4 data nodes) with:
- 16 GB of RAM allocated to elastic (out of 32)
- 8 cores
- 300 GB HDD (two nodes with SSD, two nodes with mechanical)
We are about to replace the nodes with new machines.
Looking at getting 5 used ProLiant DL360e Gen8 machines with following spec
- 2x Xeon E5-2450L 8-Core 1.8 GHz
- 64 GB RAM (allocate 32 GB to elastic)
- 500 GB SSD (6G SATA)
Our index has 12 shards, 1 replica, refresh interval 5 seconds. Index size is ~500GB (including replicas) over 45M documents. Average document size is 6Kb.
The mapping contains mostly not analyzed fields and a few fields with analyzed text in which we search with several match_phrase queries (which got slow).
We are continuously updating and indexing documents using bulk at a rate of about 15k/minute.
What would your recommendations be for our setup considering that we expect to get more data (1-2M per day) and also need better search speed.