I have following cluster configuration: 3 data nodes 3 master nodes 2 client nodes.
On one of my data node i have got this exception while indexing.
[2019-02-03T18:10:45,242][WARN ][o.e.g.DanglingIndicesState] [dV6Mim8] [[logstash-2019.02.01/hsts0oIXSb6XLkdbnmPNwQ]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-02-03T18:10:45,242][WARN ][o.e.g.DanglingIndicesState] [dV6Mim8] [[.kibana/lPmP_ppGQKO6kkipJTAIgg]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
On one of my masternode i have got this exception while indexing.
azureuser@k8s-master-16305772-0:/$ kubectl logs es-master-1835410912-r3xst -n logging
[2019-02-03T17:59:59,693][INFO ][o.e.p.PluginsService ] [0HFZAxv] loaded module [transport-netty4]
[2019-02-03T17:59:59,694][INFO ][o.e.p.PluginsService ] [0HFZAxv] no plugins loaded
[2019-02-03T17:59:59,906][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
[2019-02-03T18:00:02,084][INFO ][o.e.n.Node ] initialized
[2019-02-03T18:00:02,085][INFO ][o.e.n.Node ] [0HFZAxv] starting ...
[2019-02-03T18:00:02,206][WARN ][i.n.u.i.MacAddressUtil ] Failed to find a usable hardware address from the network interfaces; using random bytes: 7b:c4:bc:70:05:36:96:50
[2019-02-03T18:00:02,311][INFO ][o.e.t.TransportService ] [0HFZAxv] publish_address {10.64.12.250:9300}, bound_addresses {10.64.12.250:9300}
[2019-02-03T18:00:02,319][INFO ][o.e.b.BootstrapChecks ] [0HFZAxv] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2019-02-03T18:00:32,356][WARN ][o.e.n.Node ] [0HFZAxv] timed out while waiting for initial discovery state - timeout: 30s
[2019-02-03T18:00:32,357][INFO ][o.e.n.Node ] [0HFZAxv] started
[2019-02-03T18:01:53,258][INFO ][o.e.c.s.ClusterService ] [0HFZAxv] detected_master {9FXPreu}{9FXPreuSTKqC5pT3RPLNnw}{HAjE2zxcRA-Kgt29kPfsig}{10.64.12.13}{10.64.12.13:9300}, added {{eXo1izK}{eXo1izKCRUKbvwSKHQ8ThQ}{P6Lxx90eTJWzGf6MDzUHuA}{10.64.13.179}{10.64.13.179:9300},{L6DePRd}{L6DePRdBTuGmByyeFW8KEg}{BRDmCYBZRiigQCpvUynZwA}{10.64.13.10}{10.64.13.10:9300},{9FXPreu}{9FXPreuSTKqC5pT3RPLNnw}{HAjE2zxcRA-Kgt29kPfsig}{10.64.12.13}{10.64.12.13:9300},{uza2FFn}{uza2FFnbRc2wW1mIF0Tzug}{lZiQJLdGTjOWGgRaYWQ8Eg}{10.64.12.189}{10.64.12.189:9300},{SsaTPo7}{SsaTPo7ATOKCOpmq9hlyjw}{Zu3xgeIjQImu_bz4XCIyUw}{10.64.12.12}{10.64.12.12:9300},{JtSOm5S}{JtSOm5SaTze33yh0kHEXUw}{tVn2OtdLSXKCX6_fQGp9mg}{10.64.12.197}{10.64.12.197:9300},}, reason: zen-disco-receive(from master [master {9FXPreu}{9FXPreuSTKqC5pT3RPLNnw}{HAjE2zxcRA-Kgt29kPfsig}{10.64.12.13}{10.64.12.13:9300} committed version [608]])
[2019-02-03T18:01:56,190][WARN ][o.e.c.NodeConnectionsService] [0HFZAxv] failed to connect to node {L6DePRd}{L6DePRdBTuGmByyeFW8KEg}{BRDmCYBZRiigQCpvUynZwA}{10.64.13.10}{10.64.13.10:9300} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [L6DePRd][10.64.13.10:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:370) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:495) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:460) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:305) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-5.2.0.jar:5.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: Host is unreachable: 10.64.13.10/10.64.13.10:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
One of the Client pod having following issue.
[2019-02-03T18:08:27,148][INFO ][o.e.p.PluginsService ] [8zf3fC0] no plugins loaded
[2019-02-03T18:08:27,319][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
[2019-02-03T18:08:29,152][INFO ][o.e.n.Node ] initialized
[2019-02-03T18:08:29,152][INFO ][o.e.n.Node ] [8zf3fC0] starting ...
[2019-02-03T18:08:29,240][WARN ][i.n.u.i.MacAddressUtil ] Failed to find a usable hardware address from the network interfaces; using random bytes: 23:32:71:70:bb:95:d3:2a
[2019-02-03T18:08:29,324][INFO ][o.e.t.TransportService ] [8zf3fC0] publish_address {10.64.12.190:9300}, bound_addresses {10.64.12.190:9300}
[2019-02-03T18:08:29,330][INFO ][o.e.b.BootstrapChecks ] [8zf3fC0] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2019-02-03T18:08:37,648][INFO ][o.e.c.s.ClusterService ] [8zf3fC0] detected_master {9FXPreu}{9FXPreuSTKqC5pT3RPLNnw}{HAjE2zxcRA-Kgt29kPfsig}{10.64.12.13}{10.64.12.13:9300}, added {{0HFZAxv}{0HFZAxvWSUG-ob6bMBWMSg}{SKmqqcbPQSi25_aB0fVbkw}{10.64.12.250}{10.64.12.250:9300},{dV6Mim8}{dV6Mim8eRBiPb6DPULGmbw}{2EVILRoERQO10Pa0l7sBnA}{10.64.12.69}{10.64.12.69:9300},{JtSOm5S}{JtSOm5SaTze33yh0kHEXUw}{2LcaHocyQ_Soixj8ERDjwA}{10.64.13.15}{10.64.13.15:9300},{MlxALPm}{MlxALPmtS7-QJvKOrxrtDg}{IDTxMJAOR1Kk0HdlQSsKeQ}{10.64.13.180}{10.64.13.180:9300},{eXo1izK}{eXo1izKCRUKbvwSKHQ8ThQ}{P6Lxx90eTJWzGf6MDzUHuA}{10.64.13.179}{10.64.13.179:9300},{9FXPreu}{9FXPreuSTKqC5pT3RPLNnw}{HAjE2zxcRA-Kgt29kPfsig}{10.64.12.13}{10.64.12.13:9300},}, reason: zen-disco-receive(from master [master {9FXPreu}{9FXPreuSTKqC5pT3RPLNnw}{HAjE2zxcRA-Kgt29kPfsig}{10.64.12.13}{10.64.12.13:9300} committed version [666]])
[2019-02-03T18:08:38,023][INFO ][o.e.h.HttpServer ] [8zf3fC0] publish_address {10.64.12.190:9200}, bound_addresses {10.64.12.190:9200}
[2019-02-03T18:08:38,023][INFO ][o.e.n.Node ] [8zf3fC0] started
[2019-02-03T18:08:50,116][INFO ][o.e.c.s.ClusterService ] [8zf3fC0] added {{SsaTPo7}{SsaTPo7ATOKCOpmq9hlyjw}{2kPmpkPHR8eqYetGPkJZ-w}{10.64.12.12}{10.64.12.12:9300},}, reason: zen-disco-receive(from master [master {9FXPreu}{9FXPreuSTKqC5pT3RPLNnw}{HAjE2zxcRA-Kgt29kPfsig}{10.64.12.13}{10.64.12.13:9300} committed version [668]])