When to use multiple clusters


(Alex Kehayias) #1

I have several large indices (100M docs) on the same cluster. Is there any
advice of when it is appropriate to separate into multiple clusters vs one
large one? Each index has a slightly different usage profile (read vs write
heavy, update vs insert). How many indices would you recommend for a single
cluster? Is it ok to have many large indices on the same cluster?

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/24ebf4dc-f281-4574-8cbb-cb049c4fac71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

Depends what your hardware profiles are like, and a bunch of other things
related to you and your environment.
eg If you have high end servers then it makes sense to put your heavy
read/write indexes into a cluster on those, then leave the rest for more
average machines.

We have multiple clusters based on use. One for an application text based
search, one for application logging, one for system logging and we're going
to spin up another one for a new project we're starting. This might sound
like a waste of resources, and it probably is to a degree, but we have the
infrastructure for it and it makes things easier to manage.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 24 July 2014 00:34, Alex Kehayias alex@shareablee.com wrote:

I have several large indices (100M docs) on the same cluster. Is there any
advice of when it is appropriate to separate into multiple clusters vs one
large one? Each index has a slightly different usage profile (read vs write
heavy, update vs insert). How many indices would you recommend for a single
cluster? Is it ok to have many large indices on the same cluster?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/24ebf4dc-f281-4574-8cbb-cb049c4fac71%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/24ebf4dc-f281-4574-8cbb-cb049c4fac71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YqsEB2BC1zexHZOPp56%3DGJCaWcp72OjtNMYqZxqXXVxg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Alex Kehayias) #3

Thanks Mark! We're deploying on EC2 (always a good time). Seems like the
mixture of different indices that have different usage profiles is leading
to some performance issues that a dedicated cluster would be more
appropriate for.

On Wednesday, July 23, 2014 7:04:34 PM UTC-4, Mark Walkom wrote:

Depends what your hardware profiles are like, and a bunch of other things
related to you and your environment.
eg If you have high end servers then it makes sense to put your heavy
read/write indexes into a cluster on those, then leave the rest for more
average machines.

We have multiple clusters based on use. One for an application text based
search, one for application logging, one for system logging and we're going
to spin up another one for a new project we're starting. This might sound
like a waste of resources, and it probably is to a degree, but we have the
infrastructure for it and it makes things easier to manage.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 24 July 2014 00:34, Alex Kehayias <al...@shareablee.com <javascript:>>
wrote:

I have several large indices (100M docs) on the same cluster. Is there
any advice of when it is appropriate to separate into multiple clusters vs
one large one? Each index has a slightly different usage profile (read vs
write heavy, update vs insert). How many indices would you recommend for a
single cluster? Is it ok to have many large indices on the same cluster?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/24ebf4dc-f281-4574-8cbb-cb049c4fac71%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/24ebf4dc-f281-4574-8cbb-cb049c4fac71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6571673c-472f-4013-9608-d511a9f66d86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4