During indexing, if I specify a routing_key as 0, will it be indexed in
shardId 0? Will routing_key 3 correspond to shard Id 3? Similarly for all
other keys if I have 20 unique routing values since 0 % 20 will be 0 and 3
% 20 will be 3, etc.
There is no hash but a specific set of routing keys [0..19] = number of
shards [0..19] that I have.
Is this deterministic and documented by ES regarding this routing key to
shard Id behavior? Or is it internal to ES and changeable anytime?
Routing is deterministic, otherwise we couldn't know where data is located
when using the get API (this API goes to a single shard, not all of them).
However, you should not rely on the distribution of the hash values as this
is an implementation detail that we could indeed change at some point.
I don't know what your use-case is, but if you really need to manage the
sharding yourself, the easiest way to do it would be to creates 20 indices
with 1 shard instead of 1 index with 20 shards. I would discourage to do it
though.
During indexing, if I specify a routing_key as 0, will it be indexed in
shardId 0? Will routing_key 3 correspond to shard Id 3? Similarly for all
other keys if I have 20 unique routing values since 0 % 20 will be 0 and 3
% 20 will be 3, etc.
There is no hash but a specific set of routing keys [0..19] = number of
shards [0..19] that I have.
Is this deterministic and documented by ES regarding this routing key to
shard Id behavior? Or is it internal to ES and changeable anytime?
Thanks for the reply. That was important for me to understand.
However, I am a little concerned with your comment on the equivalence of 1
index with 20 shards and 20 indices with one shard each. You mentioned that
you would discourage the latter.
Can you please explain why? Is it for management reasons or performance
overhead reasons? I can deal with the former but not the latter unless if
you have some pointers. Thanks,
Thanks,
Sandeep
On Thursday, 28 August 2014 13:28:32 UTC+5:30, Adrien Grand wrote:
Hi Sandeep,
Routing is deterministic, otherwise we couldn't know where data is
located when using the get API (this API goes to a single shard, not all of
them). However, you should not rely on the distribution of the hash values
as this is an implementation detail that we could indeed change at some
point.
I don't know what your use-case is, but if you really need to manage the
sharding yourself, the easiest way to do it would be to creates 20 indices
with 1 shard instead of 1 index with 20 shards. I would discourage to do it
though.
On Thu, Aug 28, 2014 at 8:31 AM, 'Sandeep Ramesh Khanzode' via
elasticsearch <elasti...@googlegroups.com <javascript:>> wrote:
Hi
Say I create an index with 20 shards.
During indexing, if I specify a routing_key as 0, will it be indexed in
shardId 0? Will routing_key 3 correspond to shard Id 3? Similarly for all
other keys if I have 20 unique routing values since 0 % 20 will be 0 and 3
% 20 will be 3, etc.
There is no hash but a specific set of routing keys [0..19] = number of
shards [0..19] that I have.
Is this deterministic and documented by ES regarding this routing key to
shard Id behavior? Or is it internal to ES and changeable anytime?
However, I am a little concerned with your comment on the equivalence of 1
index with 20 shards and 20 indices with one shard each. You mentioned that
you would discourage the latter.
Can you please explain why? Is it for management reasons or performance
overhead reasons? I can deal with the former but not the latter unless if
you have some pointers. Thanks,
Sorry for the confusion, what I would like to discourage is not having 20
indices with one shard but trying to manage sharding manually instead of
relying on elasticsearch's routing mechanism that abstracts the number of
shards.
Is it possible to intercept the Search Response, on the server itself,
from the ES Server to the node hosting the TransportClient? I can use a
plugin to do that, but I have no idea which classes to extend and which
modules to register. Can you please provide some details? Thanks,
However, I am a little concerned with your comment on the equivalence of
1 index with 20 shards and 20 indices with one shard each. You mentioned
that you would discourage the latter.
Can you please explain why? Is it for management reasons or performance
overhead reasons? I can deal with the former but not the latter unless if
you have some pointers. Thanks,
Sorry for the confusion, what I would like to discourage is not having 20
indices with one shard but trying to manage sharding manually instead of
relying on elasticsearch's routing mechanism that abstracts the number of
shards.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.