I'm using elastic 1.3.1 in production atm and want to migrate all the servers to 2.1.
Running 2.1 on a test server at the moment with SSD's and 24 cores, 64GB RAM.
The index running there is around +/- 80M rows, 54GB in size, and allocated 4 shards to it. It's not that big, but query search rate is very high and I want to optimize that speed as much as possible.
I'm not 100% confident of switching everything to 2.1 just yet because there are some slowdowns I'm experiencing at the moment when doing searches.
What I'm seeing now is that there are a lot of _segments generated by elastic and that this has a significant impact on search speed, around 24 segments per shard => 100 segments in total.
When I run _optimize?max_num_segments=1 searches go a lot faster after that
- I want to keep the number of segments to the lowest as possible:
- What are the best setting "merge" settings for the index?
(knowing that the size of the merges will be relatively low and the server is running on SSD)
- Having the latest data is less crucial, so everything can be aggregated in a translog first and only merged after a while, as long as searches go fast enough.
- I know that everything in ES 2.1 is now stored as "doc_values" by default, I'm a bit afraid that some of the speed benefits are lost with this approach,
- Do you think using "fielddata" again for some field would be benefitial in terms of search speed?
- The store that is used for elastic on linux is now niofs or mmapfs, would explictely defining "index.store.type: "mmapfs" be any beneficial in my case? (answer here was to leave these settings as is as much as possible - Mmapfs vs niofs?)
- Lastly - do you see any improvements to the query posted below, searches that are done are mostly build up in that way.
Thanks a lot in advance!
Segments of index:
Settings of index:
Example query + time on subsequent runs: