# Elasticsearch v5.0.2 (master eligible) nodes as Docker container keeps on restarting **URL:** \ **Category:** Elasticsearch\ **Created:** [March 30, 2018, 3:39pm UTC](https://discuss.elastic.co/t/elasticsearch-v5-0-2-master-eligible-nodes-as-docker-container-keeps-on-restarting/126251 "2018-03-30T15:39:46Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![bintut](https://avatars.discourse-cdn.com/v4/letter/b/ea5d25/32.png) [@bintut](https://discuss.elastic.co/u/bintut)\ **Post date:** [March 30, 2018, 3:39pm UTC](https://discuss.elastic.co/t/elasticsearch-v5-0-2-master-eligible-nodes-as-docker-container-keeps-on-restarting/126251/1 "2018-03-30T15:39:46Z")
Hi all, Pardon me if my issue has been answered somewhere or is not the right place to ask. This is my first post to this forum. I built a custom Docker image based on RHEL v7.x using the Elasticsearch v5.0.2 RPM from Elastic and Oracle Server JRE v1.0.8\_161. However the master eligible nodes as Docker container keeps on restarting. Below is a sample Docker command I executed to run master eligible Docker container: ``` $ docker run -d --name master01 --hostname master01 \ --privileged=true --cap-add=IPC_LOCK --restart always \ --cpus 2.0 -m 2560M --memory-reservation 2048M --memory-swap=0 --memory-swappiness=0 \ --ulimit memlock=-1:-1 --ulimit nofile=131072:262144 --ulimit nproc=131072:262144 \ -e MALLOC_ARENA_MAX=4 -e environment=development -e cluster=elasticsearch -e cluster_type=transaction \ --add-host "coordinating01 coordinating01.example.com:192.168.210.15" \ --add-host "coordinating02 coordinating02.example.com:192.168.210.16" \ --add-host "coordinating03 coordinating03.example.com:192.168.210.17" \ --add-host "data01 data01.example.com:192.168.210.15" \ --add-host "data02 data02.example.com:192.168.210.16" \ --add-host "data03 data03.example.com:192.168.210.17" \ --add-host "ingest01 ingest01.example.com:192.168.210.15" \ --add-host "ingest02 ingest02.example.com:192.168.210.16" \ --add-host "ingest03 ingest03.example.com:192.168.210.17" \ --add-host "master01 master01.example.com:192.168.210.15" \ --add-host "master02 master02.example.com:192.168.210.16" \ --add-host "master03 master03.example.com:192.168.210.17" \ --label name=master01 --label environment=development \ -v /etc/localtime:/etc/localtime:ro \ -v /home/elasticsearch/nfs/common/elasticsearch/configs/master01.yml:/etc/elasticsearch/elasticsearch.yml:ro \ -v /home/elasticsearch/nfs/common/elasticsearch/configs/jvm.options-master:/etc/elasticsearch/jvm.options:ro \ -v /home/elasticsearch/nfs/common/elasticsearch/configs/log4j2.properties:/etc/elasticsearch/log4j2.properties:ro \ -v /home/elasticsearch/data/db/elasticsearch/master/01:/data \ -v /home/elasticsearch/nfs/data/db/elasticsearch/master/01:/app/snapshot \ -v /home/elasticsearch/nfs/data/log/elasticsearch/master/01:/var/log/elasticsearch \ -w /usr/share/elasticsearch --network host -p 9200:9200 -p 9300:9300 elasticsearch:1522210252 \ /usr/local/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.conf=/etc/elasticsearch \ -Etransport.publish_port=9300 -Ehttp.publish_port=9200 ``` Thank you in advance. ---
**Author:** ![bintut](https://avatars.discourse-cdn.com/v4/letter/b/ea5d25/32.png) [@bintut](https://discuss.elastic.co/u/bintut)\ **Post date:** [March 30, 2018, 3:42pm UTC](https://discuss.elastic.co/t/elasticsearch-v5-0-2-master-eligible-nodes-as-docker-container-keeps-on-restarting/126251/2 "2018-03-30T15:42:40Z")
Below are the current Elasticsearch master eligible node configurations: ``` [master01.yml] cluster.name: ES_Cluster node.name: master01 node.master: true node.data: false node.ingest: false index.codec: best_compression path.data: /data path.logs: /var/log/elasticsearch path.repo: ["/app/snapshot"] bootstrap.memory_lock: true network.host: _site_ transport.host: _site_ http.host: _site_ http.port: 9200 transport.tcp.port: 9300 discovery.zen.ping.unicast.hosts: ["master01:9300", "master02:9300", "master03:9300"] discovery.zen.minimum_master_nodes: 2 discovery.zen.ping_timeout: 300s discovery.zen.join_timeout: 150s discovery.zen.publish_timeout: 150s discovery.zen.fd.ping_interval: 150s discovery.zen.fd.ping_timeout: 300s discovery.zen.fd.ping_retries: 3 cluster.routing.allocation.disk.threshold_enabled: true cluster.routing.allocation.disk.watermark.low: .98 cluster.routing.allocation.disk.watermark.high: .99 xpack.security.enabled: false xpack.monitoring.enabled: false xpack.watcher.enabled: false [master02.yml] cluster.name: ES_Cluster node.name: master02 node.master: true node.data: false node.ingest: false index.codec: best_compression path.data: /data path.logs: /var/log/elasticsearch path.repo: ["/app/snapshot"] bootstrap.memory_lock: true network.host: _site_ transport.host: _site_ http.host: _site_ http.port: 9200 transport.tcp.port: 9300 discovery.zen.ping.unicast.hosts: ["master01:9300", "master02:9300", "master03:9300"] discovery.zen.minimum_master_nodes: 2 discovery.zen.ping_timeout: 300s discovery.zen.join_timeout: 150s discovery.zen.publish_timeout: 150s discovery.zen.fd.ping_interval: 150s discovery.zen.fd.ping_timeout: 300s discovery.zen.fd.ping_retries: 3 cluster.routing.allocation.disk.threshold_enabled: true cluster.routing.allocation.disk.watermark.low: .98 cluster.routing.allocation.disk.watermark.high: .99 xpack.security.enabled: false xpack.monitoring.enabled: false xpack.watcher.enabled: false [master03.yml] cluster.name: ES_Cluster node.name: master03 node.master: true node.data: false node.ingest: false index.codec: best_compression path.data: /data path.logs: /var/log/elasticsearch path.repo: ["/app/snapshot"] bootstrap.memory_lock: true network.host: _site_ transport.host: _site_ http.host: _site_ http.port: 9200 transport.tcp.port: 9300 discovery.zen.ping.unicast.hosts: ["master01:9300", "master02:9300", "master03:9300"] discovery.zen.minimum_master_nodes: 2 discovery.zen.ping_timeout: 300s discovery.zen.join_timeout: 150s discovery.zen.publish_timeout: 150s discovery.zen.fd.ping_interval: 150s discovery.zen.fd.ping_timeout: 300s discovery.zen.fd.ping_retries: 3 cluster.routing.allocation.disk.threshold_enabled: true cluster.routing.allocation.disk.watermark.low: .98 cluster.routing.allocation.disk.watermark.high: .99 xpack.security.enabled: false xpack.monitoring.enabled: false xpack.watcher.enabled: false ``` ---
**Author:** ![bintut](https://avatars.discourse-cdn.com/v4/letter/b/ea5d25/32.png) [@bintut](https://discuss.elastic.co/u/bintut)\ **Post date:** [March 30, 2018, 3:44pm UTC](https://discuss.elastic.co/t/elasticsearch-v5-0-2-master-eligible-nodes-as-docker-container-keeps-on-restarting/126251/3 "2018-03-30T15:44:35Z")
Below are the logs: ``` # rpm -qi elasticsearch Name : elasticsearch Epoch : 0 Version : 5.0.2 Release : 1 Architecture: noarch Install Date: Wed Mar 28 12:11:11 2018 Group : Application/Internet Size : 36525042 License : 2009 Signature : RSA/SHA512, Thu Nov 24 18:30:01 2016, Key ID d27d666cd88e42b4 Source RPM : elasticsearch-5.0.2-1-src.rpm Build Date : Thu Nov 24 18:09:58 2016 Build Host : packer-virtualbox-iso-1468515721 Relocations : /usr Packager : Elasticsearch Vendor : Elasticsearch URL : https://www.elastic.co/ Summary : Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html Description : Elasticsearch subproject :distribution:rpm $ java -version java version "1.8.0_161" Java(TM) SE Runtime Environment (build 1.8.0_161-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode) $ uname -r 3.10.0-693.11.6.el7.x86_64 $ docker exec master01 ls -l /usr/java/jdk1.8.0_161/jre/lib/rt.jar -rw-r--r--. 1 10 143 66309610 Dec 20 08:21 /usr/java/jdk1.8.0_161/jre/lib/rt.jar $ docker exec master01 stat /usr/java/jdk1.8.0_161/jre/lib/rt.jar File: '/usr/java/jdk1.8.0_161/jre/lib/rt.jar' Size: 66309610 Blocks: 129512 IO Block: 4096 regular file Device: fd0ch/64780d Inode: 4271129 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 10/ UNKNOWN) Gid: ( 143/ UNKNOWN) Access: 2018-03-29 12:45:41.666187269 +0800 Modify: 2017-12-20 08:21:14.000000000 +0800 Change: 2018-03-28 12:25:02.744575141 +0800 Birth: - ``` ---
**Author:** ![bintut](https://avatars.discourse-cdn.com/v4/letter/b/ea5d25/32.png) [@bintut](https://discuss.elastic.co/u/bintut)\ **Post date:** [March 30, 2018, 3:47pm UTC](https://discuss.elastic.co/t/elasticsearch-v5-0-2-master-eligible-nodes-as-docker-container-keeps-on-restarting/126251/4 "2018-03-30T15:47:16Z")
Snippet of the logs ``` [2018-03-29T13:06:39,261][INFO][o.e.b.BootstrapCheck] [master01] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks [2018-03-29T13:06:39,553][WARN][o.e.d.z.p.u.UnicastZenPing] [master01] [1] failed send ping to {#zen_unicast_2#}{Efv1VRQnRnKF8t42W3jqgw}{192.168.210.16}{192.168.210.16:9300} java.lang.IllegalStateException: handshake failed with {#zen_unicast_2#}{Efv1VRQnRnKF8t42W3jqgw}{192.168.210.16}{192.168.210.16:9300} at org.elasticsearch.transport.TransportService.handshake(TransportService.java:364) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:339) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:313) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$2.run(UnicastZenPing.java:419) [elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.0.2.jar:5.0.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [][192.168.210.16:9300][internal:transport/handshake] disconnected [2018-03-29T13:07:09,282][WARN][o.e.n.Node] [master01] timed out while waiting for initial discovery state - timeout: 30s [2018-03-29T13:07:09,293][INFO][o.e.h.HttpServer] [master01] publish_address {192.168.210.15:9200}, bound_addresses {172.17.0.1:9200}, {172.19.0.1:9200}, {192.168.210.15:9200} [2018-03-29T13:07:09,294][INFO][o.e.n.Node] [master01] started [2018-03-29T13:11:39,453][WARN][o.e.d.z.ZenDiscovery] [master01] failed to connect to master [{tp-elasticsearch-service-master-03}{DFpIqUY2Rdy2ZoS_5CIeeA}{ZxiWrmpTReqXxBxxQ9tF2g}{192.168.210.17}{192.168.210.17:9300}], retrying... org.elasticsearch.transport.ConnectTransportException: [tp-elasticsearch-service-master-03][192.168.210.17:9300] connect_timeout[30s] at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:379) ~[?:?] at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:403) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:377) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:285) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:468) [elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:415) [elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.discovery.zen.ZenDiscovery.access$3800(ZenDiscovery.java:86) [elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1187) [elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.0.2.jar:5.0.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.210.17:9300 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:627) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:513) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:467) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:437) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) ~[?:?] ... 1 more ... ... ... [2018-03-29T13:12:22,512][INFO][o.e.n.Node] [master01] started [2018-03-29T13:15:26,750][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [master01] no known master node, scheduling a retry [2018-03-29T13:15:56,753][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [master01] timed out while retrying [cluster:monitor/health] after failure (timeout [30s]) [2018-03-29T13:15:56,754][WARN][r.suppressed] path: /_cluster/health, params: {pretty=true} org.elasticsearch.discovery.MasterNotDiscoveredException at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:214) [elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:350) [elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:240) [elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.cluster.service.ClusterService$NotifyTimeout.run(ClusterService.java:957) [elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.0.2.jar:5.0.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] ``` ---
**Author:** ![bintut](https://avatars.discourse-cdn.com/v4/letter/b/ea5d25/32.png) [@bintut](https://discuss.elastic.co/u/bintut)\ **Post date:** [March 30, 2018, 3:48pm UTC](https://discuss.elastic.co/t/elasticsearch-v5-0-2-master-eligible-nodes-as-docker-container-keeps-on-restarting/126251/5 "2018-03-30T15:48:33Z")
``` /usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file /usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file /usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file [2018-03-29T15:19:25,044][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [master01] fatal error in thread [main], exiting java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: java/io/InterruptedIOException at org.elasticsearch.indices.analysis.AnalysisModule.setupAnalyzers(AnalysisModule.java:290) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.indices.analysis.AnalysisModule.(AnalysisModule.java:173) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.node.Node.(Node.java:304) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.node.Node.(Node.java:217) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:196) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:196) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.0.2.jar:5.0.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.0.2.jar:5.0.2] Caused by: java.lang.NoClassDefFoundError: java/io/InterruptedIOException at sun.misc.Resource.getBytes(Resource.java:124) ~[?:?] at java.net.URLClassLoader.defineClass(URLClassLoader.java:462) ~[?:1.8.0_161] at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0_161] at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[?:1.8.0_161] at java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[?:1.8.0_161] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161] at java.net.URLClassLoader.findClass(URLClassLoader.java:361) ~[?:1.8.0_161] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_161] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) ~[?:?] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_161] ... 14 more 2018-03-29 15:19:25,055 main ERROR Unable to write to stream /var/log/elasticsearch/ES_Cluster.log for appender rolling: org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/elasticsearch/ES_Cluster.log /usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file /usr/java/jdk1.8.0_161/jre/lib/rt.jar: error reading zip file Exception: java.lang.NoClassDefFoundError thrown from the UncaughtExceptionHandler in thread "main" ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 27, 2018, 3:49pm UTC](https://discuss.elastic.co/t/elasticsearch-v5-0-2-master-eligible-nodes-as-docker-container-keeps-on-restarting/126251/6 "2018-04-27T15:49:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.