John_S
March 17, 2015, 6:00pm
1
Hi All,
Is there any best practices of having on the number of shards for a
During any node failure or other events i doubts since the shards number is
Is there any metrics or formula to be done for number or shards?
Regards
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/6e51f1e4-8938-4196-84a9-007705869b6a%40googlegroups.com .https://groups.google.com/d/optout .
warkolm
March 17, 2015, 7:56pm
2
What sort of data do you have, time based or static? If it's the former
There aren't any metrics around this, other than not creating a large
On 17 March 2015 at 11:00, John S bunix1@gmail.com wrote:
Hi All,
Is there any best practices of having on the number of shards for a
During any node failure or other events i doubts since the shards number
Is there any metrics or formula to be done for number or shards?
Regards
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/6e51f1e4-8938-4196-84a9-007705869b6a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/6e51f1e4-8938-4196-84a9-007705869b6a%40googlegroups.com?utm_medium=email&utm_source=footer https://groups.google.com/d/optout .
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_B9mxm9xnJtzoSc-tj1G-MoZ7vdQ-ye%2B7woLfj7aRHJw%40mail.gmail.com .https://groups.google.com/d/optout .
andrew
March 17, 2015, 8:05pm
3
I typically suggest to start with the default of 5 shards. A single shard can hold several tens of gigabytes. Certainly in your case it seems like 20 shards is overkill for a 4 node cluster.
On Mar 17, 2015, at 11:00 AM, John S bunix1@gmail.com wrote:
Hi All,
Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.
During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...
Is there any metrics or formula to be done for number or shards?
Regards
--elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/6e51f1e4-8938-4196-84a9-007705869b6a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/6e51f1e4-8938-4196-84a9-007705869b6a%40googlegroups.com?utm_medium=email&utm_source=footer .https://groups.google.com/d/optout https://groups.google.com/d/optout .
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/DD2AA858-ABD4-49F5-9F9C-D73C01F615CE%40elastic.co .https://groups.google.com/d/optout .
My rules is : 1 primary shard per server.
Also make some estimation how big will be the single index/shard
I think it is not good if single shard exceed 10 GB, although there is no
Georgi
On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Hi All,
Is there any best practices of having on the number of shards for a
During any node failure or other events i doubts since the shards number
Is there any metrics or formula to be done for number or shards?
Regards
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com .https://groups.google.com/d/optout .
warkolm
March 18, 2015, 4:49pm
5
We recommend shards no larger than 50GB, but as you mention there is no
On 18 March 2015 at 04:09, Georgi Ivanov georgi.r.ivanov@gmail.com wrote:
My rules is : 1 primary shard per server.
Also make some estimation how big will be the single index/shard
I think it is not good if single shard exceed 10 GB, although there is no
Georgi
On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Hi All,
Is there any best practices of having on the number of shards for a
During any node failure or other events i doubts since the shards number
Is there any metrics or formula to be done for number or shards?
Regards
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com?utm_medium=email&utm_source=footer https://groups.google.com/d/optout .
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-iUMdUXTS31uHuJ8FyXJy7vNqCrP_gw0tSs1xNFkzz%3DA%40mail.gmail.com .https://groups.google.com/d/optout .
andrew
March 18, 2015, 5:02pm
6
As Mark said, there is no hard limit on how big a single shard can be, but just so it’s clear, 10GB is actually quite small for a single shard. It’s not at all uncommon for me to see shards with upwards of 60 GB or more.
On Mar 18, 2015, at 4:09 AM, Georgi Ivanov georgi.r.ivanov@gmail.com wrote:
My rules is : 1 primary shard per server.
Also make some estimation how big will be the single index/shard
I think it is not good if single shard exceed 10 GB, although there is no exact limit.
Georgi
On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Is there any best practices of having on the number of shards for a cluster? I have a 4 node cluster and used shards of 20.
During any node failure or other events i doubts since the shards number is high, replication to new node is taking more time...
Is there any metrics or formula to be done for number or shards?
Regards
--elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com?utm_medium=email&utm_source=footer .https://groups.google.com/d/optout https://groups.google.com/d/optout .
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/39389779-986A-4933-89BD-7B842B168EA0%40elastic.co .https://groups.google.com/d/optout .
Andrej
March 18, 2015, 5:29pm
7
Hi Mark,
may I ask what the reason for this recommendation is?
Thanks,
Am Mittwoch, 18. März 2015 17:50:09 UTC+1 schrieb Mark Walkom:
We recommend shards no larger than 50GB, but as you mention there is no
On 18 March 2015 at 04:09, Georgi Ivanov <georgi....@gmail.com
My rules is : 1 primary shard per server.
Also make some estimation how big will be the single index/shard
I think it is not good if single shard exceed 10 GB, although there is no
Georgi
On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Hi All,
Is there any best practices of having on the number of shards for a
During any node failure or other events i doubts since the shards number
Is there any metrics or formula to be done for number or shards?
Regards
--elasticsearc...@googlegroups.com <javascript:>.https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com?utm_medium=email&utm_source=footer https://groups.google.com/d/optout .
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/385c959b-0132-4d9d-95cd-b4b08fca9c94%40googlegroups.com .https://groups.google.com/d/optout .
warkolm
March 18, 2015, 8:39pm
8
Part of it is based on knowledge picked up from our customers, part of it
There is also a ~2 billion hard limit for documents in a single shard,
On 18 March 2015 at 10:29, Andrej Rosenheinrich <andrej.rosenheinrich@unister.de > wrote:
Hi Mark,
may I ask what the reason for this recommendation is?
Thanks,
Am Mittwoch, 18. März 2015 17:50:09 UTC+1 schrieb Mark Walkom:
We recommend shards no larger than 50GB, but as you mention there is no
On 18 March 2015 at 04:09, Georgi Ivanov georgi....@gmail.com wrote:
My rules is : 1 primary shard per server.
Also make some estimation how big will be the single index/shard
I think it is not good if single shard exceed 10 GB, although there is
Georgi
On Tuesday, March 17, 2015 at 7:00:23 PM UTC+1, John S wrote:
Hi All,
Is there any best practices of having on the number of shards for a
During any node failure or other events i doubts since the shards
Is there any metrics or formula to be done for number or shards?
Regards
--elasticsearc...@googlegroups.com .https://groups.google.com/d/ 40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/36ef3ed0-870f-41a5-915b-fb3ad919f7a0%40googlegroups.com?utm_medium=email&utm_source=footer https://groups.google.com/d/optout .
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/385c959b-0132-4d9d-95cd-b4b08fca9c94%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/385c959b-0132-4d9d-95cd-b4b08fca9c94%40googlegroups.com?utm_medium=email&utm_source=footer
For more options, visit https://groups.google.com/d/optout .
--elasticsearch+unsubscribe@googlegroups.com .https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ajDqiyn04JXqH9RcBU8vaOLKZkEdFRYnNWFiV0ihw9Q%40mail.gmail.com .https://groups.google.com/d/optout .