Upgraded from 1.4 to 2.2, now my connection fails from Java code

Elastic Stack Elasticsearch
1

I have a simple method that creates my Client object, using either the Node or Transport classes, depending where the client is running in the network. I use Node when the client is on the same physical network as the servers, and transport when it it's remote. My code is quite simple:

        final Settings settings = Settings.settingsBuilder()
                .put("cluster.name", "ih-demo")
                .put("node.name", nodeName)
                .put("http.enabled", false)
                .put("client.transport.sniff", true)
                .build();

        switch (nodeType) {
            case "node":
                NODE = nodeBuilder()
                        .settings(settings)
                        .client(true)
                        .node();

                CLIENT = NODE.client();
                break;

            case "transport":

                try {
                    CLIENT = TransportClient.builder().settings(settings).build()
                            .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("demo.starpoint.com"), 9300));
                }
                catch (UnknownHostException uhe) {
                    throw new BusinessException(uhe);
                }

                break;

            default:
                throw new BusinessException("Unhandled node type: " + nodeType);
        }

When I start my clients in transport mode, they work fine. When I start in node mode, I eventually see the following exception:

2016-03-03T11:59:39-05:00 [elasticsearch[mqconsumer][generic][T#6]] WARN org.elasticsearch.discovery.zen.ping.unicast [mqconsumer] failed to send ping to [{#z#!/bin/bash
en_unicast_4#}{127.0.0.1}{127.0.0.1:9303}]
SendRequestTransportException[[][127.0.0.1:9303][internal:discovery/zen/unicast]]; nested: NodeNotConnectedException[[][127.0.0.1:9303] Node not connected];
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:323)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPingRequestToNode(UnicastZenPing.java:440)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:426)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240)
at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106)
at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84)
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:899)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$5000(ZenDiscovery.java:75)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1260)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: NodeNotConnectedException[[][127.0.0.1:9303] Node not connected]
at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:1115)
at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:802)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:312)
... 12 more

The exception certainly is right, nothing is listening on port 9303 on localhost.

I tried googleing "elasticsearch 2.2 port 9303" but I couldn't find anything.

Any help would be greatly appreciated.

2

I solved my problem. I had to turn off network sniffing, and add all the nodes in the cluster to my settings.

        final Settings settings = Settings.settingsBuilder()
                .put("cluster.name", "ih-demo")
                .put("node.name", nodeName)
                .put("http.enabled", false)
                .put("discovery.zen.ping.multicast.enabled", false)
                .putArray("discovery.zen.ping.unicast.hosts", "demo.starpoint.com", "bender.starpoint.com")
                // .put("client.transport.sniff", true)
                .build();

The Node documentation page for 2.2 should be updated with the putArray example above (which I found in the shield docs), since multicast is is provided by a deprecated plugin.

I think the core of my problem is that my servers don't listen on localhost at, which makes the original error I was seeing make all the sense in the world.