Create new index in busy cluster

Elastic Stack Elasticsearch
1

Assuming I have a heavily loaded elasticsearch cluster, is it likely that
the performance will take a significant hit if I start to build a new index
on it?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0dcd0b9d-71a9-4ef3-8a50-1b2f468ea169%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi,

In short, yes. How much exactly depends on how intensive your indexing is.

Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Monday, April 28, 2014 11:28:41 AM UTC-4, Martin Forssen wrote:

Assuming I have a heavily loaded elasticsearch cluster, is it likely that
the performance will take a significant hit if I start to build a new index
on it?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bcf9bfd5-841d-44ef-ad20-22a71c5d065f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

On Monday, April 28, 2014 9:51:34 PM UTC+2, Otis Gospodnetic wrote:

In short, yes. How much exactly depends on how intensive your indexing is.

Our indexing is pretty intensive since we have a number of machines sending
data as fast as they can to the Elasticsearch cluster.

So now we need to think about how to mitigate this. One idea is to add a
couple of machines to the cluster and then, assuming it is possible,
control the shard allocation so that the new index we are building gets
assigned to those new nodes and none of the other indexes do. That way the
new index is isolated to a couple of machines. We would of course direct
the indexing machines to talk directly to those machines.

Would this mitigate the performance impact on the rest of the cluster?

Does such a shard allocator exist? If not, is it possible to write one as a
plugin?

/MaF

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ae101070-e2c3-4611-9a36-f380fd05534a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Yep, you can do this using

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 29 April 2014 16:56, Martin Forssen maf@recordedfuture.com wrote:

On Monday, April 28, 2014 9:51:34 PM UTC+2, Otis Gospodnetic wrote:

In short, yes. How much exactly depends on how intensive your indexing
is.

Our indexing is pretty intensive since we have a number of machines
sending data as fast as they can to the Elasticsearch cluster.

So now we need to think about how to mitigate this. One idea is to add a
couple of machines to the cluster and then, assuming it is possible,
control the shard allocation so that the new index we are building gets
assigned to those new nodes and none of the other indexes do. That way the
new index is isolated to a couple of machines. We would of course direct
the indexing machines to talk directly to those machines.

Would this mitigate the performance impact on the rest of the cluster?

Does such a shard allocator exist? If not, is it possible to write one as
a plugin?

/MaF

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ae101070-e2c3-4611-9a36-f380fd05534a%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/ae101070-e2c3-4611-9a36-f380fd05534a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624a5gtErMQ%3DfbFtfzn_C4QC-FHPor%3DkW0wwY%2B2SA_KpCtg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

On Tuesday, April 29, 2014 8:59:35 AM UTC+2, Mark Walkom wrote:

Yep, you can do this using
Elasticsearch Platform — Find real-time answers at scale | Elastic

Thanks, that's exactly what I was looking for.

/MaF

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f123731e-f0c4-4bf1-bc7f-99941bd44bbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.