Many clusters each with one small index (current) VS Single "Big index" cluster VS Multi clusters each with 3 small indexes

Hi there,

As we started a using the ES before 2 years, that time, we have created
multiple clusters where each cluster with single index.
Current model:

  1. Number Clusters: 10
  2. Number of index in cluster: 1
  3. Number of shards per cluster: 5
  4. Number of Replica: 1
  5. Per index size: 50 GB
  6. Number of nodes per cluster: 3 nodes (master and data)
  7. All these in 3 big severs.

As we use each nodes both as master and data nodes, the master gets high
loaded and if one master goes down, the master also goes down faster. The
end result is cluster becomes unavailable.

To resolve it, we are thinking following solutions:

  1. Add more hardware, use dedicated master and data nodes, increase the
    data nodes considering the future data growth and have less maintenance
    impact if a cluster goes down. (or)

  2. Merge the clusters like each cluster with 3 indexes with 3 dedicated
    master and enough number data nodes say 8. So, per cluster it may 3 index
    ie 15 primary and 15 replica shards balanced in 3 master and 8 to 10 data
    nodes. (or)

  3. Create one bigger cluster move all 10 index i.e. 50 primary and 50
    replica shards, have the cluster with 3-5 master, 20 - 30 data nodes.

To have less down time and less management effort as well supporting future
data growth which one will be the better option to go?

Any input will be highly appreciated.

Thanks much.
Muthusamy J

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/130066f7-d3f2-4de6-9306-03aa697c0d63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ES scales horizontally, so you should consider one cluster of many nodes
and multiples indexes rather than many clusters. This will also save on
management overhead.

Some other points; Set shard count to an increment of node count, you have
3 nodes so use 3/6/9/etc shards, this ensures you have balanced allocation
and allows you to easily add more nodes and continue that even allocation
(if you use >3 shards obviously).

On 7 November 2014 17:27, Muthusamy Jayaraman jaymuthus@gmail.com wrote:

Hi there,

As we started a using the ES before 2 years, that time, we have created
multiple clusters where each cluster with single index.
Current model:

  1. Number Clusters: 10
  2. Number of index in cluster: 1
  3. Number of shards per cluster: 5
  4. Number of Replica: 1
  5. Per index size: 50 GB
  6. Number of nodes per cluster: 3 nodes (master and data)
  7. All these in 3 big severs.

As we use each nodes both as master and data nodes, the master gets high
loaded and if one master goes down, the master also goes down faster. The
end result is cluster becomes unavailable.

To resolve it, we are thinking following solutions:

  1. Add more hardware, use dedicated master and data nodes, increase the
    data nodes considering the future data growth and have less maintenance
    impact if a cluster goes down. (or)

  2. Merge the clusters like each cluster with 3 indexes with 3 dedicated
    master and enough number data nodes say 8. So, per cluster it may 3 index
    ie 15 primary and 15 replica shards balanced in 3 master and 8 to 10 data
    nodes. (or)

  3. Create one bigger cluster move all 10 index i.e. 50 primary and 50
    replica shards, have the cluster with 3-5 master, 20 - 30 data nodes.

To have less down time and less management effort as well supporting
future data growth which one will be the better option to go?

Any input will be highly appreciated.

Thanks much.
Muthusamy J

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/130066f7-d3f2-4de6-9306-03aa697c0d63%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/130066f7-d3f2-4de6-9306-03aa697c0d63%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DoAVEujVORbrc-Vr0ihDSTgsGVVBYF0USDfO0WKsMa0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.