Any issues using 2 shards for an index?

Elastic Stack Elasticsearch
1

Hi All,

We are thinking of using two shards per index + 1 replica to keep the number of shards low for some indices. Are there any gotchas with using 2 shards per index besides that at most we can scale the writes to this index to two machines?

Thanks,

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8C50A329-FB87-45C8-A38E-B908756183E0%40venarc.com.
For more options, visit https://groups.google.com/d/optout.

2

Writes and reads :slight_smile:
You may also end up with some nodes holding more, smaller shards than
others, which will mean uneven load.

If you have potential for many small indexes, check out routing as an
alternative.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 July 2014 12:16, Drew Kutcharian drew@venarc.com wrote:

Hi All,

We are thinking of using two shards per index + 1 replica to keep the
number of shards low for some indices. Are there any gotchas with using 2
shards per index besides that at most we can scale the writes to this index
to two machines?

Thanks,

Drew

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8C50A329-FB87-45C8-A38E-B908756183E0%40venarc.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YWOMcfRHqF%2B9jpvPs6H8OqmPx78GJbb-vr1z43r-32fg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks Mark. I know we can scale the reads by adding more replicas. Also the issue with nodes containing a lot of shards can be fixed using index shard allocation.

I mainly wanted to see if there are any other "undocumented" gotchas.

On Jul 8, 2014, at 7:19 PM, Mark Walkom markw@campaignmonitor.com wrote:

Writes and reads :slight_smile:
You may also end up with some nodes holding more, smaller shards than others, which will mean uneven load.

If you have potential for many small indexes, check out routing as an alternative.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 July 2014 12:16, Drew Kutcharian drew@venarc.com wrote:
Hi All,

We are thinking of using two shards per index + 1 replica to keep the number of shards low for some indices. Are there any gotchas with using 2 shards per index besides that at most we can scale the writes to this index to two machines?

Thanks,

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8C50A329-FB87-45C8-A38E-B908756183E0%40venarc.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YWOMcfRHqF%2B9jpvPs6H8OqmPx78GJbb-vr1z43r-32fg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A3E48B75-E531-4ECD-BF14-85F8C907D23E%40venarc.com.
For more options, visit https://groups.google.com/d/optout.

4

You should be fine. We run about 1600 indexes most of which are single
shard. The are pretty low traffic so it works or fine.

Yes we know about routing, no it won't help us. 1600 isn't enough to cause
a problem.
On Jul 8, 2014 10:24 PM, "Drew Kutcharian" drew@venarc.com wrote:

Thanks Mark. I know we can scale the reads by adding more replicas. Also
the issue with nodes containing a lot of shards can be fixed using index
shard allocation.

I mainly wanted to see if there are any other “undocumented” gotchas.

On Jul 8, 2014, at 7:19 PM, Mark Walkom markw@campaignmonitor.com wrote:

Writes and reads :slight_smile:
You may also end up with some nodes holding more, smaller shards than
others, which will mean uneven load.

If you have potential for many small indexes, check out routing as an
alternative.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 July 2014 12:16, Drew Kutcharian drew@venarc.com wrote:

Hi All,

We are thinking of using two shards per index + 1 replica to keep the
number of shards low for some indices. Are there any gotchas with using 2
shards per index besides that at most we can scale the writes to this index
to two machines?

Thanks,

Drew

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8C50A329-FB87-45C8-A38E-B908756183E0%40venarc.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YWOMcfRHqF%2B9jpvPs6H8OqmPx78GJbb-vr1z43r-32fg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEM624YWOMcfRHqF%2B9jpvPs6H8OqmPx78GJbb-vr1z43r-32fg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/A3E48B75-E531-4ECD-BF14-85F8C907D23E%40venarc.com
https://groups.google.com/d/msgid/elasticsearch/A3E48B75-E531-4ECD-BF14-85F8C907D23E%40venarc.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3u5TMObF%3Dv%3DEB7EG59%2BFgBob%2Bw_ytpZoWdt1qguS4kqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Yep that is all manageable, but you may cross a point where managing that
becomes more hassle than it's worth.
Something to keep in mind.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 July 2014 12:24, Drew Kutcharian drew@venarc.com wrote:

Thanks Mark. I know we can scale the reads by adding more replicas. Also
the issue with nodes containing a lot of shards can be fixed using index
shard allocation.

I mainly wanted to see if there are any other “undocumented” gotchas.

On Jul 8, 2014, at 7:19 PM, Mark Walkom markw@campaignmonitor.com wrote:

Writes and reads :slight_smile:
You may also end up with some nodes holding more, smaller shards than
others, which will mean uneven load.

If you have potential for many small indexes, check out routing as an
alternative.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 July 2014 12:16, Drew Kutcharian drew@venarc.com wrote:

Hi All,

We are thinking of using two shards per index + 1 replica to keep the
number of shards low for some indices. Are there any gotchas with using 2
shards per index besides that at most we can scale the writes to this index
to two machines?

Thanks,

Drew

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8C50A329-FB87-45C8-A38E-B908756183E0%40venarc.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YWOMcfRHqF%2B9jpvPs6H8OqmPx78GJbb-vr1z43r-32fg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEM624YWOMcfRHqF%2B9jpvPs6H8OqmPx78GJbb-vr1z43r-32fg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/A3E48B75-E531-4ECD-BF14-85F8C907D23E%40venarc.com
https://groups.google.com/d/msgid/elasticsearch/A3E48B75-E531-4ECD-BF14-85F8C907D23E%40venarc.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YMaXQLcUoBnDyJLVot2CYVSTjhifeUGwWxADLvNQ7Ojg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

Yes, but for our usecase we need to use parent/child queries which are pretty much unfeasible to do any other way due to their limitations (can't do parent/child using multiple indices).

  • Drew

On Jul 8, 2014, at 8:17 PM, Mark Walkom markw@campaignmonitor.com wrote:

Yep that is all manageable, but you may cross a point where managing that becomes more hassle than it's worth.
Something to keep in mind.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 July 2014 12:24, Drew Kutcharian drew@venarc.com wrote:
Thanks Mark. I know we can scale the reads by adding more replicas. Also the issue with nodes containing a lot of shards can be fixed using index shard allocation.

I mainly wanted to see if there are any other "undocumented" gotchas.

On Jul 8, 2014, at 7:19 PM, Mark Walkom markw@campaignmonitor.com wrote:

Writes and reads :slight_smile:
You may also end up with some nodes holding more, smaller shards than others, which will mean uneven load.

If you have potential for many small indexes, check out routing as an alternative.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 July 2014 12:16, Drew Kutcharian drew@venarc.com wrote:
Hi All,

We are thinking of using two shards per index + 1 replica to keep the number of shards low for some indices. Are there any gotchas with using 2 shards per index besides that at most we can scale the writes to this index to two machines?

Thanks,

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8C50A329-FB87-45C8-A38E-B908756183E0%40venarc.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YWOMcfRHqF%2B9jpvPs6H8OqmPx78GJbb-vr1z43r-32fg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A3E48B75-E531-4ECD-BF14-85F8C907D23E%40venarc.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YMaXQLcUoBnDyJLVot2CYVSTjhifeUGwWxADLvNQ7Ojg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4630918D-A28B-4125-8B04-AF482A6B1904%40venarc.com.
For more options, visit https://groups.google.com/d/optout.