Disk awarnes on Indexing

PrasathRajan · February 20, 2015, 7:24am

Hi All,

Few of our nodes filled disks up to 90%. Is there any option to avoid
indexing on those disks instead of relocating shards. Because each shard
contains roughly 600G data, so relocating is costly as ours is very busy
cluster.

Thanks
Prasath Rajan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · February 20, 2015, 9:37am

You can change the watermark levels for them, but that is about it.

Also if you have 600GB shards then you have a problem.

On 20 February 2015 at 18:24, Prasanth R prasanth.sunrise@gmail.com wrote:

Hi All,

Few of our nodes filled disks up to 90%. Is there any option to avoid
indexing on those disks instead of relocating shards. Because each shard
contains roughly 600G data, so relocating is costly as ours is very busy
cluster.

Thanks
Prasath Rajan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

PrasathRajan · February 20, 2015, 10:43am

Problem?
On Feb 20, 2015 3:07 PM, "Mark Walkom" markwalkom@gmail.com wrote:

You can change the watermark levels for them, but that is about it.

Also if you have 600GB shards then you have a problem.

On 20 February 2015 at 18:24, Prasanth R prasanth.sunrise@gmail.com
wrote:

Hi All,

Few of our nodes filled disks up to 90%. Is there any option to avoid
indexing on those disks instead of relocating shards. Because each shard
contains roughly 600G data, so relocating is costly as ours is very busy
cluster.

Thanks
Prasath Rajan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR3fqG_SKZr7PK%3DT%2Bs1%2B-dqtBWDb7D%2B2VNFJ7dFxVjm0UQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

PrasathRajan · February 20, 2015, 6:30pm

Could you please elaborate about the problem?
On Feb 20, 2015 4:13 PM, "Prasanth R" prasanth.sunrise@gmail.com wrote:

Problem?
On Feb 20, 2015 3:07 PM, "Mark Walkom" markwalkom@gmail.com wrote:

You can change the watermark levels for them, but that is about it.

Also if you have 600GB shards then you have a problem.

On 20 February 2015 at 18:24, Prasanth R prasanth.sunrise@gmail.com
wrote:

Hi All,

Few of our nodes filled disks up to 90%. Is there any option to avoid
indexing on those disks instead of relocating shards. Because each shard
contains roughly 600G data, so relocating is costly as ours is very busy
cluster.

Thanks
Prasath Rajan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

nik9000 · February 20, 2015, 7:35pm

I have 30GB shards and the biggest problem I have is that they take a long
time to replicate to other machines. I believe there are memory issues for
very large shards as well but I don't know them that well.

Nik
On Feb 20, 2015 7:31 PM, "Prasanth R" prasanth.sunrise@gmail.com wrote:

Could you please elaborate about the problem?
On Feb 20, 2015 4:13 PM, "Prasanth R" prasanth.sunrise@gmail.com wrote:

Problem?
On Feb 20, 2015 3:07 PM, "Mark Walkom" markwalkom@gmail.com wrote:

You can change the watermark levels for them, but that is about it.

Also if you have 600GB shards then you have a problem.

On 20 February 2015 at 18:24, Prasanth R prasanth.sunrise@gmail.com
wrote:

Hi All,

Few of our nodes filled disks up to 90%. Is there any option to
avoid indexing on those disks instead of relocating shards. Because each
shard contains roughly 600G data, so relocating is costly as ours is very
busy cluster.

Thanks
Prasath Rajan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1vJrqzDgWUFKMo9aF7LxxxvEB7tbE6XUg7a1n4VbdbHQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · February 20, 2015, 8:57pm

You have 600GB of data to move if you want to reallocate or recover, you
have 600GB of data to query, which means you have a massive amount of data
to load into memory to do that. It's a big risk.

On 21 February 2015 at 06:35, Nikolas Everett nik9000@gmail.com wrote:

I have 30GB shards and the biggest problem I have is that they take a long
time to replicate to other machines. I believe there are memory issues for
very large shards as well but I don't know them that well.

Nik
On Feb 20, 2015 7:31 PM, "Prasanth R" prasanth.sunrise@gmail.com wrote:

Could you please elaborate about the problem?
On Feb 20, 2015 4:13 PM, "Prasanth R" prasanth.sunrise@gmail.com wrote:

Problem?
On Feb 20, 2015 3:07 PM, "Mark Walkom" markwalkom@gmail.com wrote:

You can change the watermark levels for them, but that is about it.

Also if you have 600GB shards then you have a problem.

On 20 February 2015 at 18:24, Prasanth R prasanth.sunrise@gmail.com
wrote:

Hi All,

Few of our nodes filled disks up to 90%. Is there any option to
avoid indexing on those disks instead of relocating shards. Because each
shard contains roughly 600G data, so relocating is costly as ours is very
busy cluster.

Thanks
Prasath Rajan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1vJrqzDgWUFKMo9aF7LxxxvEB7tbE6XUg7a1n4VbdbHQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1vJrqzDgWUFKMo9aF7LxxxvEB7tbE6XUg7a1n4VbdbHQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-frxFycej4YFq4uGxjOwBATV2EtjLaEu0phv0Koh%2Bs4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

PrasathRajan · February 21, 2015, 1:55pm

We have 10 Node cluster and each with 32 GB Ram allocated.. 9 indexes.. 50
shards...25TB data... Queries are roughly taking 1 to 2 minutes...facing
frequent node downs...

Any suggestion to improve?
On Feb 21, 2015 2:27 AM, "Mark Walkom" markwalkom@gmail.com wrote:

You have 600GB of data to move if you want to reallocate or recover, you
have 600GB of data to query, which means you have a massive amount of data
to load into memory to do that. It's a big risk.

On 21 February 2015 at 06:35, Nikolas Everett nik9000@gmail.com wrote:

I have 30GB shards and the biggest problem I have is that they take a
long time to replicate to other machines. I believe there are memory
issues for very large shards as well but I don't know them that well.

Nik
On Feb 20, 2015 7:31 PM, "Prasanth R" prasanth.sunrise@gmail.com wrote:

Could you please elaborate about the problem?
On Feb 20, 2015 4:13 PM, "Prasanth R" prasanth.sunrise@gmail.com
wrote:

Problem?
On Feb 20, 2015 3:07 PM, "Mark Walkom" markwalkom@gmail.com wrote:

You can change the watermark levels for them, but that is about it.

Also if you have 600GB shards then you have a problem.

On 20 February 2015 at 18:24, Prasanth R prasanth.sunrise@gmail.com
wrote:

Hi All,

Few of our nodes filled disks up to 90%. Is there any option to
avoid indexing on those disks instead of relocating shards. Because each
shard contains roughly 600G data, so relocating is costly as ours is very
busy cluster.

Thanks
Prasath Rajan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1vJrqzDgWUFKMo9aF7LxxxvEB7tbE6XUg7a1n4VbdbHQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1vJrqzDgWUFKMo9aF7LxxxvEB7tbE6XUg7a1n4VbdbHQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-frxFycej4YFq4uGxjOwBATV2EtjLaEu0phv0Koh%2Bs4Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-frxFycej4YFq4uGxjOwBATV2EtjLaEu0phv0Koh%2Bs4Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1q2aKp8qA0TpyNBG%2BpjKgnsKDdBb7_W%3DeMVPYo9HzUqw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · February 21, 2015, 8:31pm

Reduce your shard size and increase the count, add more nodes, reduce your
heap to 31GB.

On 22 February 2015 at 00:55, Prasanth R prasanth.sunrise@gmail.com wrote:

We have 10 Node cluster and each with 32 GB Ram allocated.. 9 indexes..
50 shards...25TB data... Queries are roughly taking 1 to 2 minutes...facing
frequent node downs...

Any suggestion to improve?
On Feb 21, 2015 2:27 AM, "Mark Walkom" markwalkom@gmail.com wrote:

You have 600GB of data to move if you want to reallocate or recover, you
have 600GB of data to query, which means you have a massive amount of data
to load into memory to do that. It's a big risk.

On 21 February 2015 at 06:35, Nikolas Everett nik9000@gmail.com wrote:

I have 30GB shards and the biggest problem I have is that they take a
long time to replicate to other machines. I believe there are memory
issues for very large shards as well but I don't know them that well.

Nik
On Feb 20, 2015 7:31 PM, "Prasanth R" prasanth.sunrise@gmail.com
wrote:

Could you please elaborate about the problem?
On Feb 20, 2015 4:13 PM, "Prasanth R" prasanth.sunrise@gmail.com
wrote:

Problem?
On Feb 20, 2015 3:07 PM, "Mark Walkom" markwalkom@gmail.com wrote:

You can change the watermark levels for them, but that is about it.

Also if you have 600GB shards then you have a problem.

On 20 February 2015 at 18:24, Prasanth R prasanth.sunrise@gmail.com
wrote:

Hi All,

Few of our nodes filled disks up to 90%. Is there any option to
avoid indexing on those disks instead of relocating shards. Because each
shard contains roughly 600G data, so relocating is costly as ours is very
busy cluster.

Thanks
Prasath Rajan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1vJrqzDgWUFKMo9aF7LxxxvEB7tbE6XUg7a1n4VbdbHQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1vJrqzDgWUFKMo9aF7LxxxvEB7tbE6XUg7a1n4VbdbHQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-frxFycej4YFq4uGxjOwBATV2EtjLaEu0phv0Koh%2Bs4Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-frxFycej4YFq4uGxjOwBATV2EtjLaEu0phv0Koh%2Bs4Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1q2aKp8qA0TpyNBG%2BpjKgnsKDdBb7_W%3DeMVPYo9HzUqw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1q2aKp8qA0TpyNBG%2BpjKgnsKDdBb7_W%3DeMVPYo9HzUqw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8yErB_RRhs-%2BpvrXEyb25YQNTgq41jHDOa6VDioP4izw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

PrasathRajan · February 21, 2015, 8:36pm

Thanks for the suggestion... Let me try this...
On Feb 22, 2015 2:02 AM, "Mark Walkom" markwalkom@gmail.com wrote:

Reduce your shard size and increase the count, add more nodes, reduce your
heap to 31GB.

On 22 February 2015 at 00:55, Prasanth R prasanth.sunrise@gmail.com
wrote:

We have 10 Node cluster and each with 32 GB Ram allocated.. 9 indexes..
50 shards...25TB data... Queries are roughly taking 1 to 2 minutes...facing
frequent node downs...

Any suggestion to improve?
On Feb 21, 2015 2:27 AM, "Mark Walkom" markwalkom@gmail.com wrote:

You have 600GB of data to move if you want to reallocate or recover, you
have 600GB of data to query, which means you have a massive amount of data
to load into memory to do that. It's a big risk.

On 21 February 2015 at 06:35, Nikolas Everett nik9000@gmail.com wrote:

I have 30GB shards and the biggest problem I have is that they take a
long time to replicate to other machines. I believe there are memory
issues for very large shards as well but I don't know them that well.

Nik
On Feb 20, 2015 7:31 PM, "Prasanth R" prasanth.sunrise@gmail.com
wrote:

Could you please elaborate about the problem?
On Feb 20, 2015 4:13 PM, "Prasanth R" prasanth.sunrise@gmail.com
wrote:

Problem?
On Feb 20, 2015 3:07 PM, "Mark Walkom" markwalkom@gmail.com wrote:

You can change the watermark levels for them, but that is about it.

Also if you have 600GB shards then you have a problem.

On 20 February 2015 at 18:24, Prasanth R <prasanth.sunrise@gmail.com

wrote:

Hi All,

Few of our nodes filled disks up to 90%. Is there any option to
avoid indexing on those disks instead of relocating shards. Because each
shard contains roughly 600G data, so relocating is costly as ours is very
busy cluster.

Thanks
Prasath Rajan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1vJrqzDgWUFKMo9aF7LxxxvEB7tbE6XUg7a1n4VbdbHQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1vJrqzDgWUFKMo9aF7LxxxvEB7tbE6XUg7a1n4VbdbHQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-frxFycej4YFq4uGxjOwBATV2EtjLaEu0phv0Koh%2Bs4Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-frxFycej4YFq4uGxjOwBATV2EtjLaEu0phv0Koh%2Bs4Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1q2aKp8qA0TpyNBG%2BpjKgnsKDdBb7_W%3DeMVPYo9HzUqw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1q2aKp8qA0TpyNBG%2BpjKgnsKDdBb7_W%3DeMVPYo9HzUqw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8yErB_RRhs-%2BpvrXEyb25YQNTgq41jHDOa6VDioP4izw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8yErB_RRhs-%2BpvrXEyb25YQNTgq41jHDOa6VDioP4izw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR3C5DqGS5f_JmMqKDT%2BPp9aMxbUAox25CKvmtaXuNSHiQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.