Transport Client connectedNodes() duplicates

Hi,

for testing purposes, I've started up a stand-alone Elasticsearch node on
my laptop, and am using the transport client to connect to it. When I
initialize the client using "sniff=true", and then print out the list of
connected nodes, as follows:

TransportClient client = new TransportClient(
  ImmutableSettings.builder()
    .put("client.transport.ignore_cluster_name",true)
    .put("client.transport.sniff",true)
);

client.addTransportAddress(new 

InetSocketTransportAddress("192.168.1.5",9300));

for(DiscoveryNode node: client.connectedNodes()) {
  System.out.println(node);
}

I get two nodes in the output:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]

These are the same node listed twice, presumably once as the node that I
added via "addTransportAddress()", and once as sniffed out by the sniffer.
You can tell that the latter one is more detailed, and includes the actual
node id and name, while the former is just the basic network information.
Notice also that they both refer to the same IP and port combination.

When I run the same test, but with sniff=false, I get one node, but
somewhat surprisingly, the generic "transport" node has been dropped from
the list:

[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]

I this the expected behavior ? Why does the sniffer not simply replace
the node info like the client does with sniff=false ? Why does it
essentially leave the node in there twice, apparently ignoring the fact
that there are two entries with the exact same connection info ?

I looked at the code, and apparently it dedupes nodes based on ID, which in
this case are obviously different (#transport#-1
vs 15AyTluTS4Wj26tKgkyQDA), but ignores the fact that the IP and port are
identical.

My expectation was that with sniff=true, it would replace the initial with
the one sniffed from the cluster, and then add any additional nodes it has
discovered, but in reality, I end up with 2x the nodes when I add my entire
cluster via addTransportAddress().

Thanks,

Stefan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Interesting. May be you could open an issue for that?

Something like "Transport Client with sniff duplicates nodes"?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 sept. 2014 à 04:50, Stefan Will stefan.will@gmail.com a écrit :

Hi,

for testing purposes, I've started up a stand-alone Elasticsearch node on my laptop, and am using the transport client to connect to it. When I initialize the client using "sniff=true", and then print out the list of connected nodes, as follows:

TransportClient client = new TransportClient(
  ImmutableSettings.builder()
    .put("client.transport.ignore_cluster_name",true)
    .put("client.transport.sniff",true)
);

client.addTransportAddress(new InetSocketTransportAddress("192.168.1.5",9300));

for(DiscoveryNode node: client.connectedNodes()) {
  System.out.println(node);
}

I get two nodes in the output:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]

These are the same node listed twice, presumably once as the node that I added via "addTransportAddress()", and once as sniffed out by the sniffer. You can tell that the latter one is more detailed, and includes the actual node id and name, while the former is just the basic network information. Notice also that they both refer to the same IP and port combination.

When I run the same test, but with sniff=false, I get one node, but somewhat surprisingly, the generic "transport" node has been dropped from the list:

[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]

I this the expected behavior ? Why does the sniffer not simply replace the node info like the client does with sniff=false ? Why does it essentially leave the node in there twice, apparently ignoring the fact that there are two entries with the exact same connection info ?

I looked at the code, and apparently it dedupes nodes based on ID, which in this case are obviously different (#transport#-1 vs 15AyTluTS4Wj26tKgkyQDA), but ignores the fact that the IP and port are identical.

My expectation was that with sniff=true, it would replace the initial with the one sniffed from the cluster, and then add any additional nodes it has discovered, but in reality, I end up with 2x the nodes when I add my entire cluster via addTransportAddress().

Thanks,

Stefan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E8DBD877-D828-44DF-968C-63F6EEDBB247%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

The two nodes are okay. The #transport# node connection on Mac OS X is just
using IPv6. ES enumerates all network devices so it finds out about that
connection. No need to be concerned, it can be configured if you want that,
either by JVM property, or by elasticsearch settings in config for the
network module.

Jörg

On Tue, Sep 2, 2014 at 4:50 AM, Stefan Will stefan.will@gmail.com wrote:

Hi,

for testing purposes, I've started up a stand-alone Elasticsearch node on
my laptop, and am using the transport client to connect to it. When I
initialize the client using "sniff=true", and then print out the list of
connected nodes, as follows:

TransportClient client = new TransportClient(
  ImmutableSettings.builder()
    .put("client.transport.ignore_cluster_name",true)
    .put("client.transport.sniff",true)
);

client.addTransportAddress(new

InetSocketTransportAddress("192.168.1.5",9300));

for(DiscoveryNode node: client.connectedNodes()) {
  System.out.println(node);
}

I get two nodes in the output:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

These are the same node listed twice, presumably once as the node that I
added via "addTransportAddress()", and once as sniffed out by the sniffer.
You can tell that the latter one is more detailed, and includes the actual
node id and name, while the former is just the basic network information.
Notice also that they both refer to the same IP and port combination.

When I run the same test, but with sniff=false, I get one node, but
somewhat surprisingly, the generic "transport" node has been dropped from
the list:

[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

I this the expected behavior ? Why does the sniffer not simply replace
the node info like the client does with sniff=false ? Why does it
essentially leave the node in there twice, apparently ignoring the fact
that there are two entries with the exact same connection info ?

I looked at the code, and apparently it dedupes nodes based on ID, which
in this case are obviously different (#transport#-1
vs 15AyTluTS4Wj26tKgkyQDA), but ignores the fact that the IP and port are
identical.

My expectation was that with sniff=true, it would replace the initial with
the one sniffed from the cluster, and then add any additional nodes it has
discovered, but in reality, I end up with 2x the nodes when I add my entire
cluster via addTransportAddress().

Thanks,

Stefan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNGv6mqLLRm8z4QfFqqoi1New0OP0oGiM%3D9QbZOX_%2B1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Interesting… for what it’s worth, it was indeed binding to the local IPv6 wildcard address, but publishing the external IPv4 address:

[2014-09-01 21:14:05,560][INFO ][http ] [Diablo] bound_address {inet[/0:0:0:0:0:0:0:0%0:9200]}, publish_address {inet[/192.168.1.5:9200]}

After setting "network.host: non_loopback:ipv4”, it’s now both binding to and publishing the same address:

[2014-09-02 14:18:46,988][INFO ][http ] [Hijacker] bound_address {inet[/192.168.1.5:9200]}, publish_address {inet[/192.168.1.5:9200]}

But that had no effect on the result, and the node is still duplicated:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]

[Hijacker][_KM9_mF3S96xoUx402c-dQ][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]

FWIW, for different reasons I had already disabled IPv6 on my primary interface a while ago (via networksetup -setv6off Wi-Fi).

— Stefan

On Tue, Sep 2, 2014 at 1:26 PM, joergprante@gmail.com
joergprante@gmail.com wrote:

The two nodes are okay. The #transport# node connection on Mac OS X is just
using IPv6. ES enumerates all network devices so it finds out about that
connection. No need to be concerned, it can be configured if you want that,
either by JVM property, or by elasticsearch settings in config for the
network module.
Elasticsearch Platform — Find real-time answers at scale | Elastic
Jörg
On Tue, Sep 2, 2014 at 4:50 AM, Stefan Will stefan.will@gmail.com wrote:

Hi,

for testing purposes, I've started up a stand-alone Elasticsearch node on
my laptop, and am using the transport client to connect to it. When I
initialize the client using "sniff=true", and then print out the list of
connected nodes, as follows:

TransportClient client = new TransportClient(
  ImmutableSettings.builder()
    .put("client.transport.ignore_cluster_name",true)
    .put("client.transport.sniff",true)
);

client.addTransportAddress(new

InetSocketTransportAddress("192.168.1.5",9300));

for(DiscoveryNode node: client.connectedNodes()) {
  System.out.println(node);
}

I get two nodes in the output:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

These are the same node listed twice, presumably once as the node that I
added via "addTransportAddress()", and once as sniffed out by the sniffer.
You can tell that the latter one is more detailed, and includes the actual
node id and name, while the former is just the basic network information.
Notice also that they both refer to the same IP and port combination.

When I run the same test, but with sniff=false, I get one node, but
somewhat surprisingly, the generic "transport" node has been dropped from
the list:

[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

I this the expected behavior ? Why does the sniffer not simply replace
the node info like the client does with sniff=false ? Why does it
essentially leave the node in there twice, apparently ignoring the fact
that there are two entries with the exact same connection info ?

I looked at the code, and apparently it dedupes nodes based on ID, which
in this case are obviously different (#transport#-1
vs 15AyTluTS4Wj26tKgkyQDA), but ignores the fact that the IP and port are
identical.

My expectation was that with sniff=true, it would replace the initial with
the one sniffed from the cluster, and then add any additional nodes it has
discovered, but in reality, I end up with 2x the nodes when I add my entire
cluster via addTransportAddress().

Thanks,

Stefan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/7AN6m_sK7yI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNGv6mqLLRm8z4QfFqqoi1New0OP0oGiM%3D9QbZOX_%2B1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1409682693421.17318b27%40Nodemailer.
For more options, visit https://groups.google.com/d/optout.

When I start a default ES node on Mac OSX, it opens on address "0" ("bind
to all interfaces") a server socket, so I can see this in netstat

tcp46 0 0 *.9300 . LISTEN

This means a TCP6 port (in hybrid mode with TCP4 stack compatbility) is
used. For the JVM, this will be recognized as two interfaces.

This can be demonstrated with the ListNets program at

It shows something like

Display name: en0
Name: en0
InetAddress: /fe80:0:0:0:7a31:c1ff:fed6:f350%en0
InetAddress: /192.168.1.250

Display name: lo0
Name: lo0
InetAddress: /fe80:0:0:0:0:0:0:1%lo0
InetAddress: /0:0:0:0:0:0:0:1
InetAddress: /127.0.0.1

If sniff mode sees this and ES discovery is set to IP multicast (which is
the default), it will receive a response from en0 (lo0 is not able to do
multicast) and tries to open two connections to the node on "en0", because
there are two addresses on that interface. If JVM is set to
"java.net.preferIPv4Stack=true", it simply means for ES to open two IPv4
connections to the address.

Jörg

On Tue, Sep 2, 2014 at 8:31 PM, Stefan Will stefan.will@gmail.com wrote:

Interesting… for what it’s worth, it was indeed binding to the local IPv6
wildcard address, but publishing the external IPv4 address:

[2014-09-01 21:14:05,560][INFO ][http ] [Diablo]
bound_address {inet[/0:0:0:0:0:0:0:0%0:9200]}, publish_address {inet[/
192.168.1.5:9200]}

After setting "network.host: non_loopback:ipv4”, it’s now both binding
to and publishing the same address:

[2014-09-02 14:18:46,988][INFO ][http ] [Hijacker]
bound_address {inet[/192.168.1.5:9200]}, publish_address {inet[/
192.168.1.5:9200]}

But that had no effect on the result, and the node is still duplicated:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Hijacker][_KM9_mF3S96xoUx402c-dQ][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

FWIW, for different reasons I had already disabled IPv6 on my primary
interface a while ago (via networksetup -setv6off Wi-Fi).

— Stefan

On Tue, Sep 2, 2014 at 1:26 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

The two nodes are okay. The #transport# node connection on Mac OS X is
just using IPv6. ES enumerates all network devices so it finds out about
that connection. No need to be concerned, it can be configured if you want
that, either by JVM property, or by elasticsearch settings in config for
the network module.

Elasticsearch Platform — Find real-time answers at scale | Elastic

Jörg

On Tue, Sep 2, 2014 at 4:50 AM, Stefan Will stefan.will@gmail.com
wrote:

Hi,

for testing purposes, I've started up a stand-alone Elasticsearch node
on my laptop, and am using the transport client to connect to it. When I
initialize the client using "sniff=true", and then print out the list of
connected nodes, as follows:

 TransportClient client = new TransportClient(
  ImmutableSettings.builder()
    .put("client.transport.ignore_cluster_name",true)
    .put("client.transport.sniff",true)
);

client.addTransportAddress(new

InetSocketTransportAddress("192.168.1.5",9300));

for(DiscoveryNode node: client.connectedNodes()) {
  System.out.println(node);
}

I get two nodes in the output:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

These are the same node listed twice, presumably once as the node that I
added via "addTransportAddress()", and once as sniffed out by the sniffer.
You can tell that the latter one is more detailed, and includes the actual
node id and name, while the former is just the basic network information.
Notice also that they both refer to the same IP and port combination.

When I run the same test, but with sniff=false, I get one node, but
somewhat surprisingly, the generic "transport" node has been dropped from
the list:

[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

I this the expected behavior ? Why does the sniffer not simply replace
the node info like the client does with sniff=false ? Why does it
essentially leave the node in there twice, apparently ignoring the fact
that there are two entries with the exact same connection info ?

I looked at the code, and apparently it dedupes nodes based on ID, which
in this case are obviously different (#transport#-1
vs 15AyTluTS4Wj26tKgkyQDA), but ignores the fact that the IP and port are
identical.

My expectation was that with sniff=true, it would replace the initial
with the one sniffed from the cluster, and then add any additional nodes it
has discovered, but in reality, I end up with 2x the nodes when I add my
entire cluster via addTransportAddress().

Thanks,

Stefan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/7AN6m_sK7yI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNGv6mqLLRm8z4QfFqqoi1New0OP0oGiM%3D9QbZOX_%2B1g%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNGv6mqLLRm8z4QfFqqoi1New0OP0oGiM%3D9QbZOX_%2B1g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1409682693421.17318b27%40Nodemailer
https://groups.google.com/d/msgid/elasticsearch/1409682693421.17318b27%40Nodemailer?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH1RH6VNpzT1LUtfqjPnyj%2BvNQYD%3DSjfOTETc_E-hP_mw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

But I’m using unicast… ("discovery.zen.ping.multicast: false”), on both the server, and the client. Also, after setting "network.host: non_loopback:ipv4”, the ES node is no longer binding to the wildcard (0), but directly to my local LAN address. I can confirm using netstat:

tcp4 0 0 192.168.1.5.9300 . LISTEN

Since I disabled IPv6, /en0 only has a single IPv4 address on my machine (based on ListNets):

Display name: en0

Name: en0

InetAddress: /192.168.1.5

Even with all this, the transport (not node) client still gives me a duplicate node. The same behavior happens on Ubuntu as well by the way.

— Stefan

BTW - here’s the configuration of my ES server:

cluster.name: development

discovery.zen.ping.multicast: false

network.host: non_loopback:ipv4

index.number_of_shards: 3

index.number_of_replicas: 0

script.disable_dynamic: true

and this is how I initialized my client:

TransportClient client = new TransportClient(

  ImmutableSettings.builder()

    .put("client.transport.ignore_cluster_name",true)

    .put("client.transport.sniff",true)

    .put("discovery.zen.ping.multicast",false)

);

On Tue, Sep 2, 2014 at 3:31 PM, joergprante@gmail.com
joergprante@gmail.com wrote:

When I start a default ES node on Mac OSX, it opens on address "0" ("bind
to all interfaces") a server socket, so I can see this in netstat
tcp46 0 0 *.9300 . LISTEN
This means a TCP6 port (in hybrid mode with TCP4 stack compatbility) is
used. For the JVM, this will be recognized as two interfaces.
This can be demonstrated with the ListNets program at
Listing Network Interface Addresses (The Java™ Tutorials > Custom Networking > Programmatic Access to Network Parameters)
It shows something like
Display name: en0
Name: en0
InetAddress: /fe80:0:0:0:7a31:c1ff:fed6:f350%en0
InetAddress: /192.168.1.250
Display name: lo0
Name: lo0
InetAddress: /fe80:0:0:0:0:0:0:1%lo0
InetAddress: /0:0:0:0:0:0:0:1
InetAddress: /127.0.0.1
If sniff mode sees this and ES discovery is set to IP multicast (which is
the default), it will receive a response from en0 (lo0 is not able to do
multicast) and tries to open two connections to the node on "en0", because
there are two addresses on that interface. If JVM is set to
"java.net.preferIPv4Stack=true", it simply means for ES to open two IPv4
connections to the address.
Jörg
On Tue, Sep 2, 2014 at 8:31 PM, Stefan Will stefan.will@gmail.com wrote:

Interesting… for what it’s worth, it was indeed binding to the local IPv6
wildcard address, but publishing the external IPv4 address:

[2014-09-01 21:14:05,560][INFO ][http ] [Diablo]
bound_address {inet[/0:0:0:0:0:0:0:0%0:9200]}, publish_address {inet[/
192.168.1.5:9200]}

After setting "network.host: non_loopback:ipv4”, it’s now both binding
to and publishing the same address:

[2014-09-02 14:18:46,988][INFO ][http ] [Hijacker]
bound_address {inet[/192.168.1.5:9200]}, publish_address {inet[/
192.168.1.5:9200]}

But that had no effect on the result, and the node is still duplicated:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Hijacker][_KM9_mF3S96xoUx402c-dQ][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

FWIW, for different reasons I had already disabled IPv6 on my primary
interface a while ago (via networksetup -setv6off Wi-Fi).

— Stefan

On Tue, Sep 2, 2014 at 1:26 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

The two nodes are okay. The #transport# node connection on Mac OS X is
just using IPv6. ES enumerates all network devices so it finds out about
that connection. No need to be concerned, it can be configured if you want
that, either by JVM property, or by elasticsearch settings in config for
the network module.

Elasticsearch Platform — Find real-time answers at scale | Elastic

Jörg

On Tue, Sep 2, 2014 at 4:50 AM, Stefan Will stefan.will@gmail.com
wrote:

Hi,

for testing purposes, I've started up a stand-alone Elasticsearch node
on my laptop, and am using the transport client to connect to it. When I
initialize the client using "sniff=true", and then print out the list of
connected nodes, as follows:

 TransportClient client = new TransportClient(
  ImmutableSettings.builder()
    .put("client.transport.ignore_cluster_name",true)
    .put("client.transport.sniff",true)
);

client.addTransportAddress(new

InetSocketTransportAddress("192.168.1.5",9300));

for(DiscoveryNode node: client.connectedNodes()) {
  System.out.println(node);
}

I get two nodes in the output:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

These are the same node listed twice, presumably once as the node that I
added via "addTransportAddress()", and once as sniffed out by the sniffer.
You can tell that the latter one is more detailed, and includes the actual
node id and name, while the former is just the basic network information.
Notice also that they both refer to the same IP and port combination.

When I run the same test, but with sniff=false, I get one node, but
somewhat surprisingly, the generic "transport" node has been dropped from
the list:

[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

I this the expected behavior ? Why does the sniffer not simply replace
the node info like the client does with sniff=false ? Why does it
essentially leave the node in there twice, apparently ignoring the fact
that there are two entries with the exact same connection info ?

I looked at the code, and apparently it dedupes nodes based on ID, which
in this case are obviously different (#transport#-1
vs 15AyTluTS4Wj26tKgkyQDA), but ignores the fact that the IP and port are
identical.

My expectation was that with sniff=true, it would replace the initial
with the one sniffed from the cluster, and then add any additional nodes it
has discovered, but in reality, I end up with 2x the nodes when I add my
entire cluster via addTransportAddress().

Thanks,

Stefan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/7AN6m_sK7yI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNGv6mqLLRm8z4QfFqqoi1New0OP0oGiM%3D9QbZOX_%2B1g%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNGv6mqLLRm8z4QfFqqoi1New0OP0oGiM%3D9QbZOX_%2B1g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1409682693421.17318b27%40Nodemailer
https://groups.google.com/d/msgid/elasticsearch/1409682693421.17318b27%40Nodemailer?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/7AN6m_sK7yI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH1RH6VNpzT1LUtfqjPnyj%2BvNQYD%3DSjfOTETc_E-hP_mw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1409689393290.7d4059fe%40Nodemailer.
For more options, visit https://groups.google.com/d/optout.

Hmmmm that's weird, never was aware of that... I always thought it was a
hybrid tcp46 port issue. Will check if it is the same on my linux machines.

Jörg

On Tue, Sep 2, 2014 at 10:23 PM, Stefan Will stefan.will@gmail.com wrote:

But I’m using unicast… ("discovery.zen.ping.multicast: false”), on both
the server, and the client. Also, after setting "network.host:
non_loopback:ipv4”, the ES node is no longer binding to the wildcard (0),
but directly to my local LAN address. I can confirm using netstat:

tcp4 0 0 192.168.1.5.9300 .
LISTEN

Since I disabled IPv6, /en0 only has a single IPv4 address on my machine
(based on ListNets):

Display name: en0
Name: en0
InetAddress: /192.168.1.5

Even with all this, the transport (not node) client still gives me a
duplicate node. The same behavior happens on Ubuntu as well by the way.

— Stefan

BTW - here’s the configuration of my ES server:

cluster.name: development
discovery.zen.ping.multicast: false
network.host: non_loopback:ipv4
index.number_of_shards: 3
index.number_of_replicas: 0
script.disable_dynamic: true

and this is how I initialized my client:

 TransportClient client = new TransportClient(
  ImmutableSettings.builder()
    .put("client.transport.ignore_cluster_name",true)
    .put("client.transport.sniff",true)
    .put("discovery.zen.ping.multicast",false)
);

On Tue, Sep 2, 2014 at 3:31 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

When I start a default ES node on Mac OSX, it opens on address "0"
("bind to all interfaces") a server socket, so I can see this in netstat

tcp46 0 0 *.9300 . LISTEN

This means a TCP6 port (in hybrid mode with TCP4 stack compatbility) is
used. For the JVM, this will be recognized as two interfaces.

This can be demonstrated with the ListNets program at
Listing Network Interface Addresses (The Java™ Tutorials > Custom Networking > Programmatic Access to Network Parameters)

It shows something like

Display name: en0
Name: en0
InetAddress: /fe80:0:0:0:7a31:c1ff:fed6:f350%en0
InetAddress: /192.168.1.250

Display name: lo0
Name: lo0
InetAddress: /fe80:0:0:0:0:0:0:1%lo0
InetAddress: /0:0:0:0:0:0:0:1
InetAddress: /127.0.0.1

If sniff mode sees this and ES discovery is set to IP multicast (which is
the default), it will receive a response from en0 (lo0 is not able to do
multicast) and tries to open two connections to the node on "en0", because
there are two addresses on that interface. If JVM is set to
"java.net.preferIPv4Stack=true", it simply means for ES to open two IPv4
connections to the address.

Jörg

On Tue, Sep 2, 2014 at 8:31 PM, Stefan Will stefan.will@gmail.com
wrote:

Interesting… for what it’s worth, it was indeed binding to the local
IPv6 wildcard address, but publishing the external IPv4 address:

[2014-09-01 21:14:05,560][INFO ][http ] [Diablo]
bound_address {inet[/0:0:0:0:0:0:0:0%0:9200]}, publish_address {inet[/
192.168.1.5:9200]}

After setting "network.host: non_loopback:ipv4”, it’s now both binding
to and publishing the same address:

[2014-09-02 14:18:46,988][INFO ][http ] [Hijacker]
bound_address {inet[/192.168.1.5:9200]}, publish_address {inet[/
192.168.1.5:9200]}

But that had no effect on the result, and the node is still duplicated:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Hijacker][_KM9_mF3S96xoUx402c-dQ][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

FWIW, for different reasons I had already disabled IPv6 on my primary
interface a while ago (via networksetup -setv6off Wi-Fi).

— Stefan

On Tue, Sep 2, 2014 at 1:26 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

The two nodes are okay. The #transport# node connection on Mac OS X
is just using IPv6. ES enumerates all network devices so it finds out about
that connection. No need to be concerned, it can be configured if you want
that, either by JVM property, or by elasticsearch settings in config for
the network module.

Elasticsearch Platform — Find real-time answers at scale | Elastic

Jörg

On Tue, Sep 2, 2014 at 4:50 AM, Stefan Will stefan.will@gmail.com
wrote:

Hi,

for testing purposes, I've started up a stand-alone Elasticsearch node
on my laptop, and am using the transport client to connect to it. When I
initialize the client using "sniff=true", and then print out the list of
connected nodes, as follows:

 TransportClient client = new TransportClient(
  ImmutableSettings.builder()
    .put("client.transport.ignore_cluster_name",true)
    .put("client.transport.sniff",true)
);

client.addTransportAddress(new

InetSocketTransportAddress("192.168.1.5",9300));

for(DiscoveryNode node: client.connectedNodes()) {
  System.out.println(node);
}

I get two nodes in the output:

[#transport#-1][Stefans-MacBook-Pro.local][inet[/192.168.1.5:9300]]
[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

These are the same node listed twice, presumably once as the node that
I added via "addTransportAddress()", and once as sniffed out by the
sniffer. You can tell that the latter one is more detailed, and includes
the actual node id and name, while the former is just the basic network
information. Notice also that they both refer to the same IP and port
combination.

When I run the same test, but with sniff=false, I get one node, but
somewhat surprisingly, the generic "transport" node has been dropped from
the list:

[Diablo][15AyTluTS4Wj26tKgkyQDA][Stefans-MacBook-Pro.local][inet[/
192.168.1.5:9300]]

I this the expected behavior ? Why does the sniffer not simply
replace the node info like the client does with sniff=false ? Why does it
essentially leave the node in there twice, apparently ignoring the fact
that there are two entries with the exact same connection info ?

I looked at the code, and apparently it dedupes nodes based on ID,
which in this case are obviously different (#transport#-1
vs 15AyTluTS4Wj26tKgkyQDA), but ignores the fact that the IP and port are
identical.

My expectation was that with sniff=true, it would replace the initial
with the one sniffed from the cluster, and then add any additional nodes it
has discovered, but in reality, I end up with 2x the nodes when I add my
entire cluster via addTransportAddress().

Thanks,

Stefan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0f7ca562-8a91-469f-98d3-08e8a27d4ea1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/7AN6m_sK7yI/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNGv6mqLLRm8z4QfFqqoi1New0OP0oGiM%3D9QbZOX_%2B1g%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNGv6mqLLRm8z4QfFqqoi1New0OP0oGiM%3D9QbZOX_%2B1g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1409682693421.17318b27%40Nodemailer
https://groups.google.com/d/msgid/elasticsearch/1409682693421.17318b27%40Nodemailer?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/7AN6m_sK7yI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH1RH6VNpzT1LUtfqjPnyj%2BvNQYD%3DSjfOTETc_E-hP_mw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH1RH6VNpzT1LUtfqjPnyj%2BvNQYD%3DSjfOTETc_E-hP_mw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1409689393290.7d4059fe%40Nodemailer
https://groups.google.com/d/msgid/elasticsearch/1409689393290.7d4059fe%40Nodemailer?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFM6Lb75Y%3DZZo8aRmjx78CMJHPR04xEpTUKEKOoYQ7V6Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.