Pardon me if my issue has been answered somewhere or is not the right place to ask. This is my first post to this forum.
I built a custom Docker image based on RHEL v7.x using the Elasticsearch v5.0.2 RPM from Elastic and Oracle Server JRE v1.0.8_161. However the master eligible nodes as Docker container keeps on restarting.
Below is a sample Docker command I executed to run master eligible Docker container:
[2018-03-29T13:06:39,261][INFO ][o.e.b.BootstrapCheck ] [master01] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-03-29T13:06:39,553][WARN ][o.e.d.z.p.u.UnicastZenPing] [master01] [1] failed send ping to {#zen_unicast_2#}{Efv1VRQnRnKF8t42W3jqgw}{192.168.210.16}{192.168.210.16:9300}
java.lang.IllegalStateException: handshake failed with {#zen_unicast_2#}{Efv1VRQnRnKF8t42W3jqgw}{192.168.210.16}{192.168.210.16:9300}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:364) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:339) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:313) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$2.run(UnicastZenPing.java:419) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.0.2.jar:5.0.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [][192.168.210.16:9300][internal:transport/handshake] disconnected
[2018-03-29T13:07:09,282][WARN ][o.e.n.Node ] [master01] timed out while waiting for initial discovery state - timeout: 30s
[2018-03-29T13:07:09,293][INFO ][o.e.h.HttpServer ] [master01] publish_address {192.168.210.15:9200}, bound_addresses {172.17.0.1:9200}, {172.19.0.1:9200}, {192.168.210.15:9200}
[2018-03-29T13:07:09,294][INFO ][o.e.n.Node ] [master01] started
[2018-03-29T13:11:39,453][WARN ][o.e.d.z.ZenDiscovery ] [master01] failed to connect to master [{tp-elasticsearch-service-master-03}{DFpIqUY2Rdy2ZoS_5CIeeA}{ZxiWrmpTReqXxBxxQ9tF2g}{192.168.210.17}{192.168.210.17:9300}], retrying...
org.elasticsearch.transport.ConnectTransportException: [tp-elasticsearch-service-master-03][192.168.210.17:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:379) ~[?:?]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:403) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:377) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:285) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:468) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:415) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.discovery.zen.ZenDiscovery.access$3800(ZenDiscovery.java:86) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1187) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.0.2.jar:5.0.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.210.17:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:627) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:513) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:467) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:437) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) ~[?:?]
... 1 more
...
...
...
[2018-03-29T13:12:22,512][INFO ][o.e.n.Node ] [master01] started
[2018-03-29T13:15:26,750][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [master01] no known master node, scheduling a retry
[2018-03-29T13:15:56,753][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [master01] timed out while retrying [cluster:monitor/health] after failure (timeout [30s])
[2018-03-29T13:15:56,754][WARN ][r.suppressed ] path: /_cluster/health, params: {pretty=true}
org.elasticsearch.discovery.MasterNotDiscoveredException
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:214) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:350) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:240) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cluster.service.ClusterService$NotifyTimeout.run(ClusterService.java:957) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.0.2.jar:5.0.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
/usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file
/usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file
/usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file
[2018-03-29T15:19:25,044][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [master01] fatal error in thread [main], exiting
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: java/io/InterruptedIOException
at org.elasticsearch.indices.analysis.AnalysisModule.setupAnalyzers(AnalysisModule.java:290) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.indices.analysis.AnalysisModule.<init>(AnalysisModule.java:173) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.node.Node.<init>(Node.java:304) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.node.Node.<init>(Node.java:217) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:196) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:196) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.0.2.jar:5.0.2]
Caused by: java.lang.NoClassDefFoundError: java/io/InterruptedIOException
at sun.misc.Resource.getBytes(Resource.java:124) ~[?:?]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:462) ~[?:1.8.0_161]
at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0_161]
at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[?:1.8.0_161]
at java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[?:1.8.0_161]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
at java.net.URLClassLoader.findClass(URLClassLoader.java:361) ~[?:1.8.0_161]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_161]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_161]
... 14 more
2018-03-29 15:19:25,055 main ERROR Unable to write to stream /var/log/elasticsearch/ES_Cluster.log for appender rolling: org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/elasticsearch/ES_Cluster.log
/usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file
/usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file
Exception: java.lang.NoClassDefFoundError thrown from the UncaughtExceptionHandler in thread "main"
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.