What will happen when shard size is too large

Elastic Stack Elasticsearch
1

I read some where that max recommended shard size is about 50G.
What are biggest bad consequences when shard size is too large.
I have, for some indice, shard size more than 100G on a jvm with 29G fixed
heap size. I suspect this is one factor that caused OutofMemory problem.
However I don't know if large shard size will results in memory issue. I
can only think of longer seek time.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ce84aba7-67a5-4484-ac93-e86cc3f7384b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Because you have to load data from the shard when you get a query, so the
larger the shard the more data you load, and OOM or slower response times
happen.

It also helps recovery and reallocation if they are smaller.

On 5 January 2015 at 13:13, Jinyuan Zhou zhou.jinyuan@gmail.com wrote:

I read some where that max recommended shard size is about 50G.
What are biggest bad consequences when shard size is too large.
I have, for some indice, shard size more than 100G on a jvm with 29G
fixed heap size. I suspect this is one factor that caused OutofMemory
problem.
However I don't know if large shard size will results in memory issue. I
can only think of longer seek time.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ce84aba7-67a5-4484-ac93-e86cc3f7384b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce84aba7-67a5-4484-ac93-e86cc3f7384b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-trGqU0u%3DCVcO8puRcDW-EcgdwLtGHd8EojgbHChVq2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks Mark,
I understand the benefit for recovery and allocation. But could you help
me a bit more about how much shards data needs to be loaded into heap at
once when doing query? I thought it is the segments that need to be
loaded/examined. If size of segment is limited and you load segment one
at time (or you don't need load all segments at once ) in a shard, then it
seems it is possible that large shard size may not have to increase the
memory footprint.
Regards,
Jack

Jinyuan (Jack) Zhou

On Sun, Jan 4, 2015 at 7:53 PM, Mark Walkom markwalkom@gmail.com wrote:

Because you have to load data from the shard when you get a query, so the
larger the shard the more data you load, and OOM or slower response times
happen.

It also helps recovery and reallocation if they are smaller.

On 5 January 2015 at 13:13, Jinyuan Zhou zhou.jinyuan@gmail.com wrote:

I read some where that max recommended shard size is about 50G.
What are biggest bad consequences when shard size is too large.
I have, for some indice, shard size more than 100G on a jvm with 29G
fixed heap size. I suspect this is one factor that caused OutofMemory
problem.
However I don't know if large shard size will results in memory issue. I
can only think of longer seek time.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ce84aba7-67a5-4484-ac93-e86cc3f7384b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce84aba7-67a5-4484-ac93-e86cc3f7384b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/xzcSH6d0QY4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-trGqU0u%3DCVcO8puRcDW-EcgdwLtGHd8EojgbHChVq2w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-trGqU0u%3DCVcO8puRcDW-EcgdwLtGHd8EojgbHChVq2w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANBTPCH0EfJBD0EtGz1Nd3k3tGQGUp%3DNfqu9xq5XNKj343Eoyw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.