Hi There,
I was running ES 2.4 with cluster 3 nodes. The cluster is working till this day my ES service is Service Unavailable and I caught log below.
My ES conifig
cluster.name: production
node.name: ${HOSTNAME}
node.master: true
node.data: true
path.data: /home/elasticsearch/data
path.logs: /home/elasticsearch/logs
bootstrap.mlockall: true
network.host: [p2p1, local]
network.bind_host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["elk", "elk-node1", "elk-node2"]
discovery.zen.ping.multicast.enabled: false
discovery.zen.minimum_master_nodes: 2
Log: I was cut some line out because the message over 5000 words.
ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];]
[2016-09-16 12:56:54,875][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
[2016-09-16 12:56:54,876][WARN ][rest.suppressed ] path: /_bulk, params: {}
ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];]
[2016-09-16 12:56:54,879][WARN ][rest.suppressed ] path: /_bulk, params: {}
ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];]
[2016-09-16 12:56:54,879][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
[2016-09-16 12:56:54,885][INFO ][node ] [elk] stopped
[2016-09-16 12:56:54,885][INFO ][node ] [elk] closing ...
[2016-09-16 12:56:54,887][INFO ][node ] [elk] closed
[2016-09-16 12:56:55,726][INFO ][node ] [elk] version[2.4.0], pid[4598], build[ce9f0c7/2016-08-29T09:14:17Z]
[2016-09-16 12:56:55,727][INFO ][node ] [elk] initializing ...
[2016-09-16 12:56:56,091][INFO ][plugins ] [elk] modules [lang-groovy, reindex, lang-expression], plugins [hq, head], sites [hq, head]
[2016-09-16 12:56:56,107][INFO ][env ] [elk] using [1] data paths, mounts [[/home (/dev/mapper/centos-home)]], net usable_space [382.1gb], net total_space [411.1gb], spins? [possibly], types [xfs]
[2016-09-16 12:56:56,107][INFO ][env ] [elk] heap size [2.9gb], compressed ordinary object pointers [true]
[2016-09-16 12:56:59,300][INFO ][node ] [elk] initialized
[2016-09-16 12:56:59,300][INFO ][node ] [elk] starting ...
[2016-09-16 12:56:59,351][INFO ][transport ] [elk] publish_address {172.16.2.215:9300}, bound_addresses {[::]:9300}
[2016-09-16 12:56:59,355][INFO ][discovery ] [elk] production/KABl_ms3RGqOfX_e9yffqA
[2016-09-16 12:57:29,357][WARN ][discovery ] [elk] waited for 30s and no initial state was set by the discovery
[2016-09-16 12:57:29,367][INFO ][http ] [elk] publish_address {172.16.2.215:9200}, bound_addresses {[::]:9200}
[2016-09-16 12:57:29,368][INFO ][node ] [elk] started
[2016-09-16 12:57:31,063][DEBUG][action.admin.indices.create] [elk] no known master node, scheduling a retry
[2016-09-16 12:57:31,063][DEBUG][action.admin.indices.create] [elk] no known master node, scheduling a retry
[2016-09-16 12:57:31,063][DEBUG][action.admin.indices.create] [elk] no known master node, scheduling a retry
[2016-09-16 12:57:31,064][DEBUG][action.admin.indices.create] [elk] no known master node, scheduling a retry
Please help me to get the cluster up.