Hi,
I ran into some performance problems with elasticsearch. We are at the development/test stage.
We used the "kopf" plugin and it doesn't work proper anymore and some api calls like "_cat/nodes" run into a timeout.
Our Setup is a Cluster with 3 dedicated master nodes, 4 client nodes (2 for imports and 2 for searches) and 9 Data Nodes.
Data node hardware:
8 cpus , 8-16 Gb mem and we use ssd's (10 GB each)
We allready migratet 2 Years of costumer documents and have about 170 Millionen documents and about 1,5 TB used space.
We store our documents per user and time, e.g. costumer_2015_05 for one index with one shard( 12 indices per year and costumer) and set an alias for every costumer for better search performance. Because there is no difference to make an indice with 12 shards per year or make 12 indices with one shard (always 12 lucene instances).
The _cluster/health api call looks fine too:
{
"cluster_name" : "our clustername",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 17,
"number_of_data_nodes" : 9,
"active_primary_shards" : 9418,
"active_shards" : 18836,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
Now the questions:
How can we make our Cluster better and avoid timeouts with apicalls?
Should we upgrade our memory to 16 or 32 Gb for better performance?
Is there another free plugin that works better than kopf?
We'd like stay with our architecture with 1 indice per month and costumer, since the alias searches went well.
Regards
Andi