I am setting up a L-E-K solution to be able to compute statistics about our transactions.
I prepared a VM running logstash instances pulling from redis queues and rerouting structured logs to a elasticsearch 0.90.9 cluster.
We have more or less 15k Tps and we expect to grow (double up).
The indexes follow a template deployed on all nodes.
All nodes can be master.
All nodes contain data.
The problem is... The system becomes slow...
Daily indexes with this setup grow up to 20/30gb each, reaching easily 1billion of documents in 7/10days.
The cluster is composed by 4 * 24 core 96gb RAM systems.
I could add some "powerful" VM (8 core 32gb ram each) without data (we are limited to 40gb disk space).
Should I increase the number of shards and keep replicas to 1?
Since I am continuously feeding the es cluster, maybe would be useful to exploit some no-data nodes to load balance search queries.
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/11591431-490c-4656-b82d-ed038c2aa26f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.