Setup of a cluster

Elastic Stack Elasticsearch
1

ello,
I am setting up a L-E-K solution to be able to compute statistics about our transactions.
I prepared a VM running logstash instances pulling from redis queues and rerouting structured logs to a elasticsearch 0.90.9 cluster.
We have more or less 15k Tps and we expect to grow (double up).
The indexes follow a template deployed on all nodes.

All nodes can be master.
All nodes contain data.

The problem is... The system becomes slow...

Daily indexes with this setup grow up to 20/30gb each, reaching easily 1billion of documents in 7/10days.

The cluster is composed by 4 * 24 core 96gb RAM systems.

I could add some "powerful" VM (8 core 32gb ram each) without data (we are limited to 40gb disk space).

Should I increase the number of shards and keep replicas to 1?

Since I am continuously feeding the es cluster, maybe would be useful to exploit some no-data nodes to load balance search queries.

Any advice?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/11591431-490c-4656-b82d-ed038c2aa26f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

When you say slow, do you mean queries are slow? If so, can you give some
example queries and also response times that you are seeing?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8f142603-0ac5-4210-b3ec-53399d0bb4c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

I am using Kibana, so typical dashboards with 6-7 facets on potentially non
low cardinality fields.

Slow, I mean non reactive dashboard, taking several seconds (20 or more) to
make the full dashboard appear.

Il giorno giovedì 27 marzo 2014 13:38:34 UTC+1, Binh Ly ha scritto:

When you say slow, do you mean queries are slow? If so, can you give some
example queries and also response times that you are seeing?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1a8ee179-8069-4df6-bfe6-71734643fb45%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.