Hello,
i have a problem with nodes connections. When i start them everything seems OK and then i try to send to master some data and getting this error. I think problem is at master node but do not know exactly where. I have tried a lot of things but nothing helps.
Log from Master:
[2019-08-05T14:14:15,731][INFO ][o.e.c.r.a.AllocationService] [sm-controller-comm] Cluster health status changed from [GREEN] to [YELLOW] (reason: [{es-datanode1-comm}{UTSjPxnLTfGjYL8uZoJ8VQ}{dfUtR2K3QXu8Qsi8g_8jLg}{10.60.34.15}{10
.60.34.15:9300} left]).
[2019-08-05T14:14:15,732][INFO ][o.e.c.s.ClusterService ] [sm-controller-comm] removed {{es-datanode1-comm}{UTSjPxnLTfGjYL8uZoJ8VQ}{dfUtR2K3QXu8Qsi8g_8jLg}{10.60.34.15}{10.60.34.15:9300},}, reason: zen-disco-node-left({es-datanod
e1-comm}{UTSjPxnLTfGjYL8uZoJ8VQ}{dfUtR2K3QXu8Qsi8g_8jLg}{10.60.34.15}{10.60.34.15:9300}), reason(left)[{es-datanode1-comm}{UTSjPxnLTfGjYL8uZoJ8VQ}{dfUtR2K3QXu8Qsi8g_8jLg}{10.60.34.15}{10.60.34.15:9300} left]
[2019-08-05T14:14:18,006][DEBUG][o.e.a.a.c.n.i.TransportNodesInfoAction] [sm-controller-comm] failed to execute on node [UTSjPxnLTfGjYL8uZoJ8VQ]
org.elasticsearch.transport.NodeNotConnectedException: [es-datanode1-comm][10.60.34.15:9300] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:548) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:524) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.15.jar:5.6.15]
elasticsearch.yml enabled rows:
node.name: ${HOSTNAME}
network.host: local,ens192
node.master: true
node.data: false
node.ingest: false
path.repo: ["/mnt/perf_data/elasticsearch_backup/"]
CURL
curl -XGET 10.60.34.12:9200/_cat/nodes
10.60.34.12 9 67 1 0.04 0.06 0.05 m * sm-controller-comm
10.60.34.16 14 99 3 0.01 0.05 0.09 d - es-datanode2-comm
10.60.34.15 14 99 3 0.11 0.06 0.08 d - es-datanode1-comm
Log (different time) data node:
[2019-08-05T15:30:36,326][INFO ][o.e.d.z.ZenDiscovery ] [es-datanode1-comm] master_left [{sm-controller-comm}{zUz8rT3bRIuf-Nc4FM3X5w}{6mZmt5xLTJS-bOr44KTNAg}{10.60.34.12}{10.60.34.12:9300}], reason [transport disconnected]
[2019-08-05T15:30:36,328][WARN ][o.e.d.z.ZenDiscovery ] [es-datanode1-comm] master left (reason = transport disconnected), current nodes: nodes:
{es-datanode1-comm}{UTSjPxnLTfGjYL8uZoJ8VQ}{50_RxNWTQBuel_uQC0zPZw}{10.60.34.15}{10.60.34.15:9300}, local
{es-datanode2-comm}{JP2GSNy9RdaXeC6iqgfT8Q}{8qWEuQoIQRiBtvYF5pll2g}{10.60.34.16}{10.60.34.16:9300}
{sm-controller-comm}{zUz8rT3bRIuf-Nc4FM3X5w}{6mZmt5xLTJS-bOr44KTNAg}{10.60.34.12}{10.60.34.12:9300}, master
[2019-08-05T15:30:36,336][WARN ][o.e.c.NodeConnectionsService] [es-datanode1-comm] failed to connect to node {sm-controller-comm}{zUz8rT3bRIuf-Nc4FM3X5w}{6mZmt5xLTJS-bOr44KTNAg}{10.60.34.12}{10.60.34.12:9300} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [sm-controller-comm][10.60.34.12:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.15.jar:5.6.15]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.15.jar:5.6.15]
Best regards,
Elastic Newbie Michal