ElasticCluster cant restart and connect with kibana


(rpoveda) #1

After cluster reboot, Kibana show me the cluster's status RED

It was working fine...

¿Anyone can helpme? Thanks


(rpoveda) #2
[2016-11-08T17:26:34,166][INFO ][o.e.n.Node               ] [submiss-elk-4] stopping ...
[2016-11-08T17:26:34,802][INFO ][o.e.n.Node               ] [submiss-elk-4] stopped
[2016-11-08T17:26:34,803][INFO ][o.e.n.Node               ] [submiss-elk-4] closing ...
[2016-11-08T17:26:34,846][INFO ][o.e.n.Node               ] [submiss-elk-4] closed
[2016-11-08T17:29:06,540][INFO ][o.e.n.Node               ] [submiss-elk-4] initializing ...
[2016-11-08T17:29:08,029][INFO ][o.e.e.NodeEnvironment    ] [submiss-elk-4] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [24.6gb], net total_space [26.4gb], spins? [unknown], types [rootfs]
[2016-11-08T17:29:08,029][INFO ][o.e.e.NodeEnvironment    ] [submiss-elk-4] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-11-08T17:29:08,070][INFO ][o.e.n.Node               ] [submiss-elk-4] version[5.0.0], pid[906], build[253032b/2016-10-26T04:37:51.531Z], OS[Linux/3.10.0-327.36.3.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_111/25.111-b15]
[2016-11-08T17:29:21,561][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [aggs-matrix-stats]
[2016-11-08T17:29:21,561][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [ingest-common]
[2016-11-08T17:29:21,562][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [lang-expression]
[2016-11-08T17:29:21,562][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [lang-groovy]
[2016-11-08T17:29:21,562][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [lang-mustache]
[2016-11-08T17:29:21,562][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [lang-painless]
[2016-11-08T17:29:21,562][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [percolator]
[2016-11-08T17:29:21,562][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [reindex]
[2016-11-08T17:29:21,562][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [transport-netty3]
[2016-11-08T17:29:21,562][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] loaded module [transport-netty4]
[2016-11-08T17:29:21,563][INFO ][o.e.p.PluginsService     ] [submiss-elk-4] no plugins loaded
[2016-11-08T17:29:37,553][INFO ][o.e.n.Node               ] [submiss-elk-4] initialized
[2016-11-08T17:29:37,553][INFO ][o.e.n.Node               ] [submiss-elk-4] starting ...
[2016-11-08T17:29:38,011][INFO ][o.e.t.TransportService   ] [submiss-elk-4] publish_address {192.168.1.84:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {[fe80::250:56ff:feb0:6b3b]:9300}, {192.168.1.84:9300}
[2016-11-08T17:29:38,017][INFO ][o.e.b.BootstrapCheck     ] [submiss-elk-4] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-11-08T17:29:39,681][WARN ][o.e.m.j.JvmGcMonitorService] [submiss-elk-4] [gc][2] overhead, spent [504ms] collecting in the last [1s]
[2016-11-08T17:29:43,274][INFO ][o.e.c.s.ClusterService   ] [submiss-elk-4] detected_master {masterYi-elk-2}{ZT8CMT_xRWSpLhEkVaYvvQ}{W8rPfyUnSgi3pZpJLpBrxw}{192.168.1.82}{192.168.1.82:9300}, added {{masterYi-elk-2}{ZT8CMT_xRWSpLhEkVaYvvQ}{W8rPfyUnSgi3pZpJLpBrxw}{192.168.1.82}{192.168.1.82:9300},{masterYi-elk-1}{lpKVPiS3R1KsECN3wVB_EQ}{PAdFu0nMThK7D5bp0ug5iA}{192.168.1.81}{192.168.1.81:9300},}, reason: zen-disco-receive(from master [master {masterYi-elk-2}{ZT8CMT_xRWSpLhEkVaYvvQ}{W8rPfyUnSgi3pZpJLpBrxw}{192.168.1.82}{192.168.1.82:9300} committed version [3]])
[2016-11-08T17:29:43,603][INFO ][o.e.h.HttpServer         ] [submiss-elk-4] publish_address {192.168.1.84:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {[fe80::250:56ff:feb0:6b3b]:9200}, {192.168.1.84:9200}
[2016-11-08T17:29:43,604][INFO ][o.e.n.Node               ] [submiss-elk-4] started
[2016-11-08T17:29:43,884][INFO ][o.e.c.s.ClusterService   ] [submiss-elk-4] added {{submiss-elk-3}{papAQeloQlKIWiH7sMOteg}{iX-NNmuLQhyY5qvheWJlMQ}{192.168.1.83}{192.168.1.83:9300},}, reason: zen-disco-receive(from master [master {masterYi-elk-2}{ZT8CMT_xRWSpLhEkVaYvvQ}{W8rPfyUnSgi3pZpJLpBrxw}{192.168.1.82}{192.168.1.82:9300} committed version [4]])

(rpoveda) #3
[2016-11-08T17:31:02,198][WARN ][r.suppressed             ] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

(rpoveda) #4
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:161) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:147) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:234) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:164) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:97) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onFailure(TransportAction.java:254) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.handleException(TransportMasterNodeAction.java:190) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:954) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.transport.TcpTransport.lambda$handleException$15(TcpTransport.java:1277) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.threadpool.ThreadPool.lambda$static$0(ThreadPool.java:147) ~[elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1275) [elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1267) [elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1211) [elasticsearch-5.0.0.jar:5.0.0]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-5.0.0.jar:5.0.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:610) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:513) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:467) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:437) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) [netty-common-4.1.5.Final.jar:4.1.5.Final]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

(rpoveda) #5
	Suppressed: org.elasticsearch.transport.RemoteTransportException: [masterYi-elk-2][192.168.1.82:9300][indices:admin/create]
	Caused by: org.elasticsearch.discovery.MasterNotDiscoveredException: ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:214) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:350) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:240) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.cluster.service.ClusterService$NotifyTimeout.run(ClusterService.java:936) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:444) ~[elasticsearch-5.0.0.jar:5.0.0]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
		at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

(rpoveda) #6
	Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
		at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:178) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:64) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:39) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:140) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:132) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:103) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:53) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:182) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.ingest.IngestActionFilter.apply(IngestActionFilter.java:82) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:180) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:153) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1348) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.threadpool.ThreadPool.lambda$static$0(ThreadPool.java:147) ~[elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1306) [elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1205) [elasticsearch-5.0.0.jar:5.0.0]
		at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-5.0.0.jar:5.0.0]

(rpoveda) #7
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372) ~[netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358) ~[netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350) ~[netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:610) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:513) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:467) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:437) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) [netty-common-4.1.5.Final.jar:4.1.5.Final]
		at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]

(rpoveda) #8

sorry for sharding, but i cant send more of 5000 chars and i cant upload text files...


(rpoveda) #9

anyone could help me??


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.