I experience slow search performance on a quite large index (we talk about 5-10 seconds for a simple term query on a long field with returns ~10k docs).
First of all some details about my cluster:
- Elasticsearch version: 2.2.0
- 4 nodes with 64GB RAM (30GB heap) and 24 cores each
- Storage: RAID 10 spinning disks (ye ... I know its not ideal)
- ~ 2,000,000,000 docs
- overall size is 500GB
- 4 Shards (replication = 1 - see below)
- Search volume is quite low but with a lot of aggregations and performance matters a lot
I initially tried 2 shards and 1 replica but the search performance was not that great. I guess that's because I then utilize only two of my 4 nodes with every query I run (which than have to search a 250GB segment).
Now I tried to do the following in order to get better utilization of resources across all 4 nodes:
- 4 shards to have one primary shard on each node
- run every search request with
preference=_primary_firstbecause I only wanted to have the replicas for failover.
I already learned that ES does not balance primary shards (only overall shards) but I manually balanced them on the right nodes (with the relocate API) for the purpose of testing this.
So now I have one primary and one replica shard on every node.
When I run a search with
preference=_primary_first it was still quite slow and I noticed using
hot_threads that still two threads run searches on the same node.
preference setting really work and is it intented to be used for such a case? Does someone have experience with that?
Now when I set the replication factor to 0 with 4 shards, I get great performance (500-3000ms uncached). Exactly what I need.
I could theoretically just run with 0 replication in Production because the data is already mirrored on the RAID.
Practically I don't think its a good idea because from time to time I run into GC issues and have to restart nodes.
Does anyone have an idea how I can achieve running single queries across all 4 nodes and still have failover replica shards in place?