Transport client sniff - AWS


(Pat Christopher) #1

Hello,
Version: 0.16.2, client and server

I'm trying to use the transport client's sniff while accessing a
cluster. I'm hitting the following two issues:

  1. when running with the cluster in the cloud (AWS ec2), I am unable
    to connect with client.transport.sniff set to true. there is no
    exception thrown at runtime, it doesn't populate the nodelist
    if I remove this setting, it connects as expected.
    if I run with the cluster running locally, my client connects and
    lists all nodes in the cluster

do I need to specify something specifically for sniff to work in a
cloud env?

  1. running with sniff set to true does not provide a durable
    connection if the discovery node is killed
    if I connect to a running cluster with sniff set to true, wait for
    the client to find all nodes in the cluster then terminate the
    original node, the client loses its connection. I expected the
    connection to fail over to one of the sniffed addresses.
    if I list two nodes explicitly, I can terminate either of the two
    listed nodes and remain connected to the cluster
I'm not sure if the Transport.Client is supposed to remain

connected if the single discovery node is termed, but it seems like it
could.

Thanks,
Pat


(Shay Banon) #2

When using sniff, the address used to connect to the nodes being added is the "publish_address" (for transport) they start with (while with a non sniff option, its the address you specify). Maybe there is a problem with the publish address that got decided for the nodes (you can see it logged when a node starts up).

Yea, sniff will always rely on the seed of addresses you specify, and if all nodes are down, then nothing will be "sniffed". It can be enhanced to try and use sniffed nodes in this case, and continue to try and figure out the cluster. Shouldn't be too difficult to implement if you are up to it (check TransportClientNodesService). pull requests are welcomed :slight_smile:

-shay.banon

On Friday, June 10, 2011 at 2:54 AM, Pat Christopher wrote:

Hello,
Version: 0.16.2, client and server

I'm trying to use the transport client's sniff while accessing a
cluster. I'm hitting the following two issues:

  1. when running with the cluster in the cloud (AWS ec2), I am unable
    to connect with client.transport.sniff set to true. there is no
    exception thrown at runtime, it doesn't populate the nodelist
    if I remove this setting, it connects as expected.
    if I run with the cluster running locally, my client connects and
    lists all nodes in the cluster

do I need to specify something specifically for sniff to work in a
cloud env?

  1. running with sniff set to true does not provide a durable
    connection if the discovery node is killed
    if I connect to a running cluster with sniff set to true, wait for
    the client to find all nodes in the cluster then terminate the
    original node, the client loses its connection. I expected the
    connection to fail over to one of the sniffed addresses.
    if I list two nodes explicitly, I can terminate either of the two
    listed nodes and remain connected to the cluster

I'm not sure if the Transport.Client is supposed to remain
connected if the single discovery node is termed, but it seems like it
could.

Thanks,
Pat


(Pat Christopher) #3

ES is binding the publish_address to the AWS internal ip, which I
can't connect to from my local machine. I'll try my next test from an
AWS node.

On my list! I'll take a look at the TransportClientNodesService
tomorrow.

Thanks,
Pat

On Jun 9, 5:23 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

When using sniff, the address used to connect to the nodes being added is the "publish_address" (for transport) they start with (while with a non sniff option, its the address you specify). Maybe there is a problem with the publish address that got decided for the nodes (you can see it logged when a node starts up).

Yea, sniff will always rely on the seed of addresses you specify, and if all nodes are down, then nothing will be "sniffed". It can be enhanced to try and use sniffed nodes in this case, and continue to try and figure out the cluster. Shouldn't be too difficult to implement if you are up to it (check TransportClientNodesService). pull requests are welcomed :slight_smile:

-shay.banon

On Friday, June 10, 2011 at 2:54 AM, Pat Christopher wrote:

Hello,
Version: 0.16.2, client and server

I'm trying to use the transport client's sniff while accessing a
cluster. I'm hitting the following two issues:

  1. when running with the cluster in the cloud (AWS ec2), I am unable
    to connect with client.transport.sniff set to true. there is no
    exception thrown at runtime, it doesn't populate the nodelist
    if I remove this setting, it connects as expected.
    if I run with the cluster running locally, my client connects and
    lists all nodes in the cluster

do I need to specify something specifically for sniff to work in a
cloud env?

  1. running with sniff set to true does not provide a durable
    connection if the discovery node is killed
    if I connect to a running cluster with sniff set to true, wait for
    the client to find all nodes in the cluster then terminate the
    original node, the client loses its connection. I expected the
    connection to fail over to one of the sniffed addresses.
    if I list two nodes explicitly, I can terminate either of the two
    listed nodes and remain connected to the cluster

I'm not sure if the Transport.Client is supposed to remain
connected if the single discovery node is termed, but it seems like it
could.

Thanks,
Pat


(Shay Banon) #4

Right, that was the main reason why the non sniff option is there at all, to allow for you on the client code to control the IPs to connect to without "sniffing".

On Friday, June 10, 2011 at 3:35 AM, Pat Christopher wrote:

ES is binding the publish_address to the AWS internal ip, which I
can't connect to from my local machine. I'll try my next test from an
AWS node.

On my list! I'll take a look at the TransportClientNodesService
tomorrow.

Thanks,
Pat

On Jun 9, 5:23 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

When using sniff, the address used to connect to the nodes being added is the "publish_address" (for transport) they start with (while with a non sniff option, its the address you specify). Maybe there is a problem with the publish address that got decided for the nodes (you can see it logged when a node starts up).

Yea, sniff will always rely on the seed of addresses you specify, and if all nodes are down, then nothing will be "sniffed". It can be enhanced to try and use sniffed nodes in this case, and continue to try and figure out the cluster. Shouldn't be too difficult to implement if you are up to it (check TransportClientNodesService). pull requests are welcomed :slight_smile:

-shay.banon

On Friday, June 10, 2011 at 2:54 AM, Pat Christopher wrote:

Hello,
Version: 0.16.2, client and server

I'm trying to use the transport client's sniff while accessing a
cluster. I'm hitting the following two issues:

  1. when running with the cluster in the cloud (AWS ec2), I am unable
    to connect with client.transport.sniff set to true. there is no
    exception thrown at runtime, it doesn't populate the nodelist
    if I remove this setting, it connects as expected.
    if I run with the cluster running locally, my client connects and
    lists all nodes in the cluster

do I need to specify something specifically for sniff to work in a
cloud env?

  1. running with sniff set to true does not provide a durable
    connection if the discovery node is killed
    if I connect to a running cluster with sniff set to true, wait for
    the client to find all nodes in the cluster then terminate the
    original node, the client loses its connection. I expected the
    connection to fail over to one of the sniffed addresses.
    if I list two nodes explicitly, I can terminate either of the two
    listed nodes and remain connected to the cluster

I'm not sure if the Transport.Client is supposed to remain
connected if the single discovery node is termed, but it seems like it
could.

Thanks,
Pat


(Pat Christopher) #5

Makes sense.

I tried this again from an AWS node, all nodes in cluster found and
correctly printed out by the client.

thanks again,
Pat

On Jun 9, 5:37 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Right, that was the main reason why the non sniff option is there at all, to allow for you on the client code to control the IPs to connect to without "sniffing".

On Friday, June 10, 2011 at 3:35 AM, Pat Christopher wrote:

ES is binding the publish_address to the AWS internal ip, which I
can't connect to from my local machine. I'll try my next test from an
AWS node.

On my list! I'll take a look at the TransportClientNodesService
tomorrow.

Thanks,
Pat

On Jun 9, 5:23 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

When using sniff, the address used to connect to the nodes being added is the "publish_address" (for transport) they start with (while with a non sniff option, its the address you specify). Maybe there is a problem with the publish address that got decided for the nodes (you can see it logged when a node starts up).

Yea, sniff will always rely on the seed of addresses you specify, and if all nodes are down, then nothing will be "sniffed". It can be enhanced to try and use sniffed nodes in this case, and continue to try and figure out the cluster. Shouldn't be too difficult to implement if you are up to it (check TransportClientNodesService). pull requests are welcomed :slight_smile:

-shay.banon

On Friday, June 10, 2011 at 2:54 AM, Pat Christopher wrote:

Hello,
Version: 0.16.2, client and server

I'm trying to use the transport client's sniff while accessing a
cluster. I'm hitting the following two issues:

  1. when running with the cluster in the cloud (AWS ec2), I am unable
    to connect with client.transport.sniff set to true. there is no
    exception thrown at runtime, it doesn't populate the nodelist
    if I remove this setting, it connects as expected.
    if I run with the cluster running locally, my client connects and
    lists all nodes in the cluster

do I need to specify something specifically for sniff to work in a
cloud env?

  1. running with sniff set to true does not provide a durable
    connection if the discovery node is killed
    if I connect to a running cluster with sniff set to true, wait for
    the client to find all nodes in the cluster then terminate the
    original node, the client loses its connection. I expected the
    connection to fail over to one of the sniffed addresses.
    if I list two nodes explicitly, I can terminate either of the two
    listed nodes and remain connected to the cluster

I'm not sure if the Transport.Client is supposed to remain
connected if the single discovery node is termed, but it seems like it
could.

Thanks,
Pat


(system) #6