I added a new node yesterday afternoon and got to a point where it should have been communicating with the cluster but its not.
I receive this error on the new node in the elasticsearch.log file:
[2016-02-25 09:33:53,802][WARN ][transport.netty ] [Node-5] Message not fully read (response) for [2698] handler org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$4@2bc2db6d, error [true], resetting
[2016-02-25 09:33:53,803][WARN ][discovery.zen.ping.unicast] [KIBANA-ALPHA] failed to send ping to [[#zen_unicast_2#][Node-5][inet[/172.16.0.4:9300]]]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:178)
I get this error on the other end (172.16.0.4):
[2016-02-25 09:34:24,545][WARN ][transport.netty ] [ELASTICSEARSH-ALPHA] exception caught on transport layer [[id: 0x9626bf34, /172.31.0.3:43864 => /172.31.0.65:9300]], closing connection
java.lang.IllegalStateException: Message not fully read (request) for requestId [2770], action [internal:discovery/zen/unicast_gte_1_4], readerIndex [59] vs expected [199]; resetting
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived
After searching I turned up this command:
curl 'localhost:9200/_nodes/jvm?pretty'
On the new node I get:
{
"cluster_name" : "ELASTICSEARCH",
"nodes" : {
"lAPD4ZLtT3Kzcj38e9hf-g" : {
"name" : "NODE-5",
"transport_address" : "inet[/172.16.0.5:9300]",
"host" : "KIBANA-ALPHA",
"ip" : "127.0.1.1",
"version" : "1.6.2",
"build" : "NA",
"http_address" : "inet[/172.16.0.5:9200]",
"attributes" : {
"data" : "false",
"master" : "false"
},
"jvm" : {
"pid" : 1801,
"version" : "1.7.0_95",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.95-b01",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1456413210741,
"mem" : {
"heap_init_in_bytes" : 268435456,
"heap_max_in_bytes" : 1065025536,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 1065025536
},
"gc_collectors" : [ "Copy", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Eden Space", "Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
}
}
}
}
On the other nodes I show:
{
"cluster_name" : "ELASTICSEARCH",
"nodes" : {
"xQXVmG3UTQKJ0lcgEjLR7g" : {
"name" : "NODE1",
"transport_address" : "172.16.0.1:9300",
"host" : "172.16.0.1",
"ip" : "172.16.0.1",
"version" : "2.2.0",
"build" : "8ff36d1",
"http_address" : "172.16.0.1:9200",
"jvm" : {
"pid" : 1490,
"version" : "1.8.0_72",
"vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
"vm_version" : "25.72-b15",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1456347785056,
"mem" : {
"heap_init_in_bytes" : 268435456,
"heap_max_in_bytes" : 1056309248,
"non_heap_init_in_bytes" : 2555904,
"non_heap_max_in_bytes" : 0,
"direct_max_in_bytes" : 1056309248
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Metaspace", "Compressed Class Space", "Par Eden Space", "Par Survivor Space", "CMS Old Gen" ],
"using_compressed_ordinary_object_pointers" : "true"
}
},
The only difference I see is the lines on transport address read "transport_address" : "inet[/172.16.0.5:9300]", on the new server and "transport_address" : "172.16.0.1:9300", on the old servers. Any help would be appreciated.
Brad