Elasticsearch.net client, endpoint strategy?

Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to my ES
cluster. I have one question I haven't been able to find the answer to. I
know that the ConnectionPool feature can check if nodes fail, but can the
client also ensure that data is written to the right shard, or does it
simply use round robin to connect?

Example:

  • A document "1234" is created
  • Based on the current number of shards, the document should be put on node
    6 and a replica on node 11.
  • Is the request sent directly to node 6, or is it sent to a random node
    which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hello Lasse ,

Following is my idea on the whole thing -

Routing -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-routing
When a index request comes , based on the ID of the request , a hash
function computes the shard to which the request has to be routed.
Hence we are achieving a load balancing based on this procedure. It has to
be also noted that the routing key can be controlled.

Sniffing - While creating a client , you can use the sniffing feature for
elasticsearch to determine all the nodes and use them all in a load
balanced way.

Thanks
Vineeth

On Fri, Sep 12, 2014 at 2:39 PM, Lasse Schou lasseschou@gmail.com wrote:

Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to my
ES cluster. I have one question I haven't been able to find the answer to.
I know that the ConnectionPool feature can check if nodes fail, but can the
client also ensure that data is written to the right shard, or does it
simply use round robin to connect?

Example:

  • A document "1234" is created
  • Based on the current number of shards, the document should be put on
    node 6 and a replica on node 11.
  • Is the request sent directly to node 6, or is it sent to a random node
    which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Alright, so the clients don't have the knowledge to route the request to
the right server. This would typically yields an extra request from the
receiving node to the node that stores the data, but on the other hand it
reduces the complexity of the client.

Can anybody confirm that this is the recommended strategy? In other
databases like Couchbase, the client knows exactly which node to route the
request to, based on a shard map that is automatically updated on regular
time intervals.

2014-09-13 5:29 GMT+02:00 vineeth mohan vm.vineethmohan@gmail.com:

Hello Lasse ,

Following is my idea on the whole thing -

Routing -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-routing
When a index request comes , based on the ID of the request , a hash
function computes the shard to which the request has to be routed.
Hence we are achieving a load balancing based on this procedure. It has to
be also noted that the routing key can be controlled.

Sniffing - While creating a client , you can use the sniffing feature for
elasticsearch to determine all the nodes and use them all in a load
balanced way.

Thanks
Vineeth

On Fri, Sep 12, 2014 at 2:39 PM, Lasse Schou lasseschou@gmail.com wrote:

Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to my
ES cluster. I have one question I haven't been able to find the answer to.
I know that the ConnectionPool feature can check if nodes fail, but can the
client also ensure that data is written to the right shard, or does it
simply use round robin to connect?

Example:

  • A document "1234" is created
  • Based on the current number of shards, the document should be put on
    node 6 and a replica on node 11.
  • Is the request sent directly to node 6, or is it sent to a random node
    which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/BpEov5aoxvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADERWXrKrdB5GnsG-YQv3W0yygyUgNzqDqxYQ4Gyr7YZ12b4ag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Yes, all REST clients (that is to exclude the native Java clients) do not
participate in the cluster and therefor don't have and shouldn't have any
knowledge of the cluster. The issue a request to a node in the cluster
which then may or may not reroute it to 1 or more nodes in an efficient
manner using the cluster meta-data it has stored locally. This is by design
and has many optimizations in place as well, and also allows you as a
client to use round-robin or make requests to a load balancer.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Sun, Sep 14, 2014 at 4:13 PM, Lasse Schou lasseschou@gmail.com wrote:

Alright, so the clients don't have the knowledge to route the request to
the right server. This would typically yields an extra request from the
receiving node to the node that stores the data, but on the other hand it
reduces the complexity of the client.

Can anybody confirm that this is the recommended strategy? In other
databases like Couchbase, the client knows exactly which node to route the
request to, based on a shard map that is automatically updated on regular
time intervals.

2014-09-13 5:29 GMT+02:00 vineeth mohan vm.vineethmohan@gmail.com:

Hello Lasse ,

Following is my idea on the whole thing -

Routing -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-routing
When a index request comes , based on the ID of the request , a hash
function computes the shard to which the request has to be routed.
Hence we are achieving a load balancing based on this procedure. It has
to be also noted that the routing key can be controlled.

Sniffing - While creating a client , you can use the sniffing feature for
elasticsearch to determine all the nodes and use them all in a load
balanced way.

Thanks
Vineeth

On Fri, Sep 12, 2014 at 2:39 PM, Lasse Schou lasseschou@gmail.com
wrote:

Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to my
ES cluster. I have one question I haven't been able to find the answer to.
I know that the ConnectionPool feature can check if nodes fail, but can the
client also ensure that data is written to the right shard, or does it
simply use round robin to connect?

Example:

  • A document "1234" is created
  • Based on the current number of shards, the document should be put on
    node 6 and a replica on node 11.
  • Is the request sent directly to node 6, or is it sent to a random node
    which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/BpEov5aoxvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CADERWXrKrdB5GnsG-YQv3W0yygyUgNzqDqxYQ4Gyr7YZ12b4ag%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CADERWXrKrdB5GnsG-YQv3W0yygyUgNzqDqxYQ4Gyr7YZ12b4ag%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtTSQdxd8q9-1-0YZLyXvDK-Cptkm%3D99T-d0_KdHqMkZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thanks for your clarifications.

2014-09-14 15:17 GMT+02:00 Itamar Syn-Hershko itamar@code972.com:

Yes, all REST clients (that is to exclude the native Java clients) do not
participate in the cluster and therefor don't have and shouldn't have any
knowledge of the cluster. The issue a request to a node in the cluster
which then may or may not reroute it to 1 or more nodes in an efficient
manner using the cluster meta-data it has stored locally. This is by design
and has many optimizations in place as well, and also allows you as a
client to use round-robin or make requests to a load balancer.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Sun, Sep 14, 2014 at 4:13 PM, Lasse Schou lasseschou@gmail.com wrote:

Alright, so the clients don't have the knowledge to route the request to
the right server. This would typically yields an extra request from the
receiving node to the node that stores the data, but on the other hand it
reduces the complexity of the client.

Can anybody confirm that this is the recommended strategy? In other
databases like Couchbase, the client knows exactly which node to route the
request to, based on a shard map that is automatically updated on regular
time intervals.

2014-09-13 5:29 GMT+02:00 vineeth mohan vm.vineethmohan@gmail.com:

Hello Lasse ,

Following is my idea on the whole thing -

Routing -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-routing
When a index request comes , based on the ID of the request , a hash
function computes the shard to which the request has to be routed.
Hence we are achieving a load balancing based on this procedure. It has
to be also noted that the routing key can be controlled.

Sniffing - While creating a client , you can use the sniffing feature
for elasticsearch to determine all the nodes and use them all in a load
balanced way.

Thanks
Vineeth

On Fri, Sep 12, 2014 at 2:39 PM, Lasse Schou lasseschou@gmail.com
wrote:

Hi,

Not sure if this is the right user group, but here goes:

I'm planning to use ElasticSearch.net as the client for connecting to
my ES cluster. I have one question I haven't been able to find the answer
to. I know that the ConnectionPool feature can check if nodes fail, but can
the client also ensure that data is written to the right shard, or does it
simply use round robin to connect?

Example:

  • A document "1234" is created
  • Based on the current number of shards, the document should be put on
    node 6 and a replica on node 11.
  • Is the request sent directly to node 6, or is it sent to a random
    node which will then forward the request to the right servers?

Thanks,
Lasse

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/52209da0-ad03-470e-b8d4-f4fd283dda4e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/BpEov5aoxvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D_-%3DcCHMJ%3DWEZQ-6FJx_xy%3DHV13UzHwj7M%2BEwpGi%2B1cA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CADERWXrKrdB5GnsG-YQv3W0yygyUgNzqDqxYQ4Gyr7YZ12b4ag%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CADERWXrKrdB5GnsG-YQv3W0yygyUgNzqDqxYQ4Gyr7YZ12b4ag%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/BpEov5aoxvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtTSQdxd8q9-1-0YZLyXvDK-Cptkm%3D99T-d0_KdHqMkZw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtTSQdxd8q9-1-0YZLyXvDK-Cptkm%3D99T-d0_KdHqMkZw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADERWXre5Ou4639T0sJ85En3Ew2hOG1j0HMpgu6RaXkWPk-P8g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.