Does Elastic Search 2.0 have shard splitting?

Elastic Stack Elasticsearch
1

I just assumed that ES was planning on building in shard splitting at some
point since it's a glaringly obvious addition to the feature set.

Then I saw this:

Users often ask why Elasticsearch doesn’t support shard-splitting — the
ability to split each shard into two or more pieces.
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/overallocation.html#id-1.8.6.8.10.2.2The
reason is that shard-splitting is a bad idea

I really hate that answer. Seems like it's just trying to excuse not
having a major feature.

The ability to incrementally add capacity to your cluster seems like a
clear win. One would even say this type of scalability is "elastic" ...

Are there plans to add this in at some future point?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bd89f8d2-6ad5-4d4e-8b8f-a0656b63afba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Why not using aliases and add more index/shards when you need to?
What is wrong with this design?

David

Le 5 déc. 2014 à 01:56, Kevin Burton burtonator@gmail.com a écrit :

I just assumed that ES was planning on building in shard splitting at some point since it's a glaringly obvious addition to the feature set.

Then I saw this:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Users often ask why Elasticsearch doesn’t support shard-splitting — the ability to split each shard into two or more pieces. The reason is that shard-splitting is a bad idea

I really hate that answer. Seems like it's just trying to excuse not having a major feature.

The ability to incrementally add capacity to your cluster seems like a clear win. One would even say this type of scalability is "elastic" ...

Are there plans to add this in at some future point?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bd89f8d2-6ad5-4d4e-8b8f-a0656b63afba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/BE60E8C8-6D36-4C50-92E1-ACE40E458D88%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

I've never found myself wanting shard splitting. I always have an analysis
update I want to apply when I want to reshard data anyway so I just scan
from one index into one with new settings.

I do find the the FAQ a bit odd though. Elasticsearch allows you to do lots
of inefficient things and that is good because flexibility is king. You
just have to understand the consequences of your actions. I find that to be
a common theme across elasticsearch, more so even than elasticity.

I figure elasticsearch.com must not think shard splitting is a competitive
advantage and none of the external contributors needs it bad enough to
implement it. I'm certainly not about to do so on the basis that one day
it might one day shave a few hours off of a task that I can run while I
sleep.

Nik
On Dec 5, 2014 1:08 AM, "David Pilato" david@pilato.fr wrote:

Why not using aliases and add more index/shards when you need to?
What is wrong with this design?

David

Le 5 déc. 2014 à 01:56, Kevin Burton burtonator@gmail.com a écrit :

I just assumed that ES was planning on building in shard splitting at some
point since it's a glaringly obvious addition to the feature set.

Then I saw this:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Users often ask why Elasticsearch doesn’t support shard-splitting —
the ability to split each shard into two or more pieces.
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/overallocation.html#id-1.8.6.8.10.2.2The
reason is that shard-splitting is a bad idea

I really hate that answer. Seems like it's just trying to excuse not
having a major feature.

The ability to incrementally add capacity to your cluster seems like a
clear win. One would even say this type of scalability is "elastic" ...

Are there plans to add this in at some future point?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bd89f8d2-6ad5-4d4e-8b8f-a0656b63afba%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bd89f8d2-6ad5-4d4e-8b8f-a0656b63afba%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BE60E8C8-6D36-4C50-92E1-ACE40E458D88%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/BE60E8C8-6D36-4C50-92E1-ACE40E458D88%40pilato.fr?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd08zLF5-dsAFUF-YELUMNy8-S001dxOZxJnQPiXcCcyfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Because too few shards is not optimal and neither is too many.

The zen is the right number of shards per node which is why shard splitting
is important.

You want the index to fit the hardware / cluster properly.

On Thursday, December 4, 2014 10:08:53 PM UTC-8, David Pilato wrote:

Why not using aliases and add more index/shards when you need to?
What is wrong with this design?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f024f914-6e0b-4fea-9566-f3e308351a8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Let say you start with a 3 nodes cluster with one index A1, 3 shards, 1 replica.
Then you feel like 3 shards is not enough. You want to add 2 new nodes and have 5 shards for your data.
Create an index A2 with 2 shards and 1 replica.

Use an alias A on top of A1, A2. You will then search into 5 shards instead of 3 initially.

Would this work for you?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 5 déc. 2014 à 18:11, Kevin Burton burtonator@gmail.com a écrit :

Because too few shards is not optimal and neither is too many.

The zen is the right number of shards per node which is why shard splitting is important.

You want the index to fit the hardware / cluster properly.

On Thursday, December 4, 2014 10:08:53 PM UTC-8, David Pilato wrote:
Why not using aliases and add more index/shards when you need to?
What is wrong with this design?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f024f914-6e0b-4fea-9566-f3e308351a8c%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/f024f914-6e0b-4fea-9566-f3e308351a8c%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/C7231EF1-4B1F-4562-80FA-E08292AD5253%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

6

That works ok if you are inserting but updates and deletes become more
complex. Scoring can get a bit funky too because your shards don't have
roughly equal frequencies. All and I'll I'd argue the adding more indecies
behind and alias is only sometimes a solution to the problem.

Nik

On Fri, Dec 5, 2014 at 12:17 PM, David Pilato david@pilato.fr wrote:

Let say you start with a 3 nodes cluster with one index A1, 3 shards, 1
replica.
Then you feel like 3 shards is not enough. You want to add 2 new nodes and
have 5 shards for your data.
Create an index A2 with 2 shards and 1 replica.

Use an alias A on top of A1, A2. You will then search into 5 shards
instead of 3 initially.

Would this work for you?

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 5 déc. 2014 à 18:11, Kevin Burton burtonator@gmail.com a écrit :

Because too few shards is not optimal and neither is too many.

The zen is the right number of shards per node which is why shard
splitting is important.

You want the index to fit the hardware / cluster properly.

On Thursday, December 4, 2014 10:08:53 PM UTC-8, David Pilato wrote:

Why not using aliases and add more index/shards when you need to?
What is wrong with this design?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f024f914-6e0b-4fea-9566-f3e308351a8c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f024f914-6e0b-4fea-9566-f3e308351a8c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/C7231EF1-4B1F-4562-80FA-E08292AD5253%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/C7231EF1-4B1F-4562-80FA-E08292AD5253%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2%3DnsHViV%2Bm1ZMcfFwr9GfFuU312GJxjJM3pUTMEceXgQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

It should be easy to export all _source fields and reindex on client side.
Because the creation of new index with new shard count and data copy should
be an atomic operation, this is non-trivial on cluster side, and difficult
to manage (cluster may run out of resources).

Jörg
Am 05.12.2014 18:40 schrieb "Nikolas Everett" nik9000@gmail.com:

That works ok if you are inserting but updates and deletes become more
complex. Scoring can get a bit funky too because your shards don't have
roughly equal frequencies. All and I'll I'd argue the adding more indecies
behind and alias is only sometimes a solution to the problem.

Nik

On Fri, Dec 5, 2014 at 12:17 PM, David Pilato david@pilato.fr wrote:

Let say you start with a 3 nodes cluster with one index A1, 3 shards, 1
replica.
Then you feel like 3 shards is not enough. You want to add 2 new nodes
and have 5 shards for your data.
Create an index A2 with 2 shards and 1 replica.

Use an alias A on top of A1, A2. You will then search into 5 shards
instead of 3 initially.

Would this work for you?

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 5 déc. 2014 à 18:11, Kevin Burton burtonator@gmail.com a écrit :

Because too few shards is not optimal and neither is too many.

The zen is the right number of shards per node which is why shard
splitting is important.

You want the index to fit the hardware / cluster properly.

On Thursday, December 4, 2014 10:08:53 PM UTC-8, David Pilato wrote:

Why not using aliases and add more index/shards when you need to?
What is wrong with this design?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f024f914-6e0b-4fea-9566-f3e308351a8c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f024f914-6e0b-4fea-9566-f3e308351a8c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/C7231EF1-4B1F-4562-80FA-E08292AD5253%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/C7231EF1-4B1F-4562-80FA-E08292AD5253%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2%3DnsHViV%2Bm1ZMcfFwr9GfFuU312GJxjJM3pUTMEceXgQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2%3DnsHViV%2Bm1ZMcfFwr9GfFuU312GJxjJM3pUTMEceXgQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG6W15x%2BY%2B9pkR8P7thxFG5_imxeyH00WySMMvEdCos4w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

8

And further, adding these new sub-indexes which compromise one composite
index is just a way to hide the creation of shards.

This is just a lazy / poor man's shard splitting and is prone to problems.
Specifically, as your cluster grows you have to re-index all your data to
rebalance.

... so this functionality is somewhat there as it is, it's just a really
lame version of shard splitting.

On Friday, December 5, 2014 11:47:07 AM UTC-8, Jörg Prante wrote:

It should be easy to export all _source fields and reindex on client side.
Because the creation of new index with new shard count and data copy should
be an atomic operation, this is non-trivial on cluster side, and difficult
to manage (cluster may run out of resources).

Jörg
Am 05.12.2014 18:40 schrieb "Nikolas Everett" <nik...@gmail.com
<javascript:>>:

That works ok if you are inserting but updates and deletes become more
complex. Scoring can get a bit funky too because your shards don't have
roughly equal frequencies. All and I'll I'd argue the adding more indecies
behind and alias is only sometimes a solution to the problem.

Nik

On Fri, Dec 5, 2014 at 12:17 PM, David Pilato <da...@pilato.fr
<javascript:>> wrote:

Let say you start with a 3 nodes cluster with one index A1, 3 shards, 1
replica.
Then you feel like 3 shards is not enough. You want to add 2 new nodes
and have 5 shards for your data.
Create an index A2 with 2 shards and 1 replica.

Use an alias A on top of A1, A2. You will then search into 5 shards
instead of 3 initially.

Would this work for you?

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 5 déc. 2014 à 18:11, Kevin Burton <burto...@gmail.com <javascript:>>
a écrit :

Because too few shards is not optimal and neither is too many.

The zen is the right number of shards per node which is why shard
splitting is important.

You want the index to fit the hardware / cluster properly.

On Thursday, December 4, 2014 10:08:53 PM UTC-8, David Pilato wrote:

Why not using aliases and add more index/shards when you need to?
What is wrong with this design?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f024f914-6e0b-4fea-9566-f3e308351a8c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f024f914-6e0b-4fea-9566-f3e308351a8c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/C7231EF1-4B1F-4562-80FA-E08292AD5253%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/C7231EF1-4B1F-4562-80FA-E08292AD5253%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2%3DnsHViV%2Bm1ZMcfFwr9GfFuU312GJxjJM3pUTMEceXgQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2%3DnsHViV%2Bm1ZMcfFwr9GfFuU312GJxjJM3pUTMEceXgQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e06b1494-43a3-4108-b3eb-f1a9b91ee3bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.