# A problem: ClusterBlockException : blocked by: \[SERVICE\_UNAVAILABLE/1/state not recovered / initialized\];

**URL:** <https://discuss.elastic.co/t/a-problem-clusterblockexception-blocked-by-service-unavailable-1-state-not-recovered-initialized/23027>\
**Category:** Elasticsearch\
**Created:** [April 1, 2015, 5:17am UTC](https://discuss.elastic.co/t/a-problem-clusterblockexception-blocked-by-service-unavailable-1-state-not-recovered-initialized/23027 "2015-04-01T05:17:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sephen\_Xu](https://avatars.discourse-cdn.com/v4/letter/s/ed8c4c/32.png) [@Sephen\_Xu](https://discuss.elastic.co/u/Sephen_Xu)\
**Post date:** [April 1, 2015, 5:17am UTC](https://discuss.elastic.co/t/a-problem-clusterblockexception-blocked-by-service-unavailable-1-state-not-recovered-initialized/23027/1 "2015-04-01T05:17:05Z")

</div>

This error occurred when i restart cluster.

Debug log:  
[2015-04-01 11:25:09,743][DEBUG][node] [es\_node\_4\_1]  
using home [/home/nat/elasticsearch-1.1.2], config  
[/home/nat/elasticsearch-1.1.2/config], data [[/home/nat/esdata.d,  
/natlog1/nat/esdata.d, /natlog4/nat/esdata.d, /natlog5/nat/esdata.d,  
/natlog6/nat/esdata.d, /natlog7/nat/esdata.d, /natlog8/nat/esdata.d]], logs  
[/home/nat/elasticsearch-1.1.2/logs], work  
[/home/nat/elasticsearch-1.1.2/work], plugins  
[/home/nat/elasticsearch-1.1.2/plugins]  
[2015-04-01 11:25:09,765][INFO][plugins] [es\_node\_4\_1]  
loaded [], sites [head, bigdesk]  
[2015-04-01 11:25:09,775][DEBUG][common.compress.lzf] using  
[UnsafeChunkDecoder] decoder  
[2015-04-01 11:25:09,787][DEBUG][env] [es\_node\_4\_1]  
using node location [[/home/nat/esdata.d/elasticsearch\_log/nodes/0,  
/natlog1/nat/esdata.d/elasticsearch\_log/nodes/0,  
/natlog4/nat/esdata.d/elasticsearch\_log/nodes/0,  
/natlog5/nat/esdata.d/elasticsearch\_log/nodes/0,  
/natlog6/nat/esdata.d/elasticsearch\_log/nodes/0,  
/natlog7/nat/esdata.d/elasticsearch\_log/nodes/0,  
/natlog8/nat/esdata.d/elasticsearch\_log/nodes/0]], local\_node\_id [0]  
[2015-04-01 11:25:09,825][INFO][node] [es\_node\_4\_2]  
version[1.1.2], pid[53749], build[e511f7b/2014-05-22T12:27:39Z]  
[2015-04-01 11:25:09,826][INFO][node] [es\_node\_4\_2]  
initializing ...  
[2015-04-01 11:25:09,826][DEBUG][node] [es\_node\_4\_2]  
using home [/home/nat/elasticsearch-1.1.2], config  
[/home/nat/elasticsearch-1.1.2/config], data [[/home/nat/esdata.d,  
/natlog2/nat/esdata.d, /natlog3/nat/esdata.d, /natlog9/nat/esdata.d,  
/natlog10/nat/esdata.d, /natlog11/nat/esdata.d, /natlog12/nat/esdata.d,  
/natlog13/nat/esdata.d]], logs [/home/nat/elasticsearch-1.1.2/logs], work  
[/home/nat/elasticsearch-1.1.2/work], plugins  
[/home/nat/elasticsearch-1.1.2/plugins]  
[2015-04-01 11:25:09,862][INFO][plugins] [es\_node\_4\_2]  
loaded [], sites [head, bigdesk]  
[2015-04-01 11:25:09,879][DEBUG][common.compress.lzf] using  
[UnsafeChunkDecoder] decoder  
[2015-04-01 11:25:09,901][DEBUG][env] [es\_node\_4\_2]  
using node location [[/home/nat/esdata.d/elasticsearch\_log/nodes/1,  
/natlog2/nat/esdata.d/elasticsearch\_log/nodes/1,  
/natlog3/nat/esdata.d/elasticsearch\_log/nodes/1,  
/natlog9/nat/esdata.d/elasticsearch\_log/nodes/1,  
/natlog10/nat/esdata.d/elasticsearch\_log/nodes/1,  
/natlog11/nat/esdata.d/elasticsearch\_log/nodes/1,  
/natlog12/nat/esdata.d/elasticsearch\_log/nodes/1,  
/natlog13/nat/esdata.d/elasticsearch\_log/nodes/1]], local\_node\_id [1]  
[2015-04-01 11:25:11,258][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [generic], type [cached], keep\_alive [30s]  
[2015-04-01 11:25:11,286][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [index], type [fixed], size [32], queue\_size [200]  
[2015-04-01 11:25:11,294][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [bulk], type [fixed], size [20], queue\_size [32]  
[2015-04-01 11:25:11,295][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [get], type [fixed], size [32], queue\_size [1k]  
[2015-04-01 11:25:11,296][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [search], type [fixed], size [96], queue\_size [1k]  
[2015-04-01 11:25:11,297][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [suggest], type [fixed], size [32], queue\_size [1k]  
[2015-04-01 11:25:11,297][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [percolate], type [fixed], size [32], queue\_size [1k]  
[2015-04-01 11:25:11,298][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [management], type [scaling], min [1], size [5],  
keep\_alive [5m]  
[2015-04-01 11:25:11,300][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [flush], type [scaling], min [1], size [5], keep\_alive  
[5m]  
[2015-04-01 11:25:11,301][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [merge], type [fixed], size [4], queue\_size [32]  
[2015-04-01 11:25:11,302][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [refresh], type [scaling], min [1], size [10],  
keep\_alive [5m]  
[2015-04-01 11:25:11,303][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [warmer], type [scaling], min [1], size [5],  
keep\_alive [5m]  
[2015-04-01 11:25:11,304][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [snapshot], type [scaling], min [1], size [5],  
keep\_alive [5m]  
[2015-04-01 11:25:11,305][DEBUG][threadpool] [es\_node\_4\_1]  
creating thread\_pool [optimize], type [fixed], size [1], queue\_size [null]  
[2015-04-01 11:25:11,348][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [generic], type [cached], keep\_alive [30s]  
[2015-04-01 11:25:11,348][DEBUG][transport.netty] [es\_node\_4\_1]  
using worker\_count[64], port[9300-9400], bind\_host[null],  
publish\_host[null], compress[false], connect\_timeout[30s],  
connections\_per\_node[2/3/6/1/1], receive\_predictor[512kb-\>512kb]  
[2015-04-01 11:25:11,357][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [index], type [fixed], size [32], queue\_size [200]  
[2015-04-01 11:25:11,362][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [bulk], type [fixed], size [20], queue\_size [32]  
[2015-04-01 11:25:11,363][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [get], type [fixed], size [32], queue\_size [1k]  
[2015-04-01 11:25:11,364][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [search], type [fixed], size [96], queue\_size [1k]  
[2015-04-01 11:25:11,364][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [suggest], type [fixed], size [32], queue\_size [1k]  
[2015-04-01 11:25:11,365][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [percolate], type [fixed], size [32], queue\_size [1k]  
[2015-04-01 11:25:11,366][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [management], type [scaling], min [1], size [5],  
keep\_alive [5m]  
[2015-04-01 11:25:11,367][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [flush], type [scaling], min [1], size [5], keep\_alive  
[5m]  
[2015-04-01 11:25:11,368][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [merge], type [fixed], size [4], queue\_size [32]  
[2015-04-01 11:25:11,368][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [refresh], type [scaling], min [1], size [10],  
keep\_alive [5m]  
[2015-04-01 11:25:11,369][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [warmer], type [scaling], min [1], size [5],  
keep\_alive [5m]  
[2015-04-01 11:25:11,370][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [snapshot], type [scaling], min [1], size [5],  
keep\_alive [5m]  
[2015-04-01 11:25:11,371][DEBUG][threadpool] [es\_node\_4\_2]  
creating thread\_pool [optimize], type [fixed], size [1], queue\_size [null]  
[2015-04-01 11:25:11,376][DEBUG][discovery.zen.ping.unicast] [es\_node\_4\_1]  
using initial hosts [10.60.9.4:9300, 10.60.9.4:9301, 10.60.9.5:9300,  
10.60.9.5:9301], with concurrent\_connects [10]  
[2015-04-01 11:25:11,380][DEBUG][discovery.zen] [es\_node\_4\_1]  
using ping.timeout [3s], master\_election.filter\_client [true],  
master\_election.filter\_data [false]  
[2015-04-01 11:25:11,381][DEBUG][discovery.zen.elect] [es\_node\_4\_1]  
using minimum\_master\_nodes [3]  
[2015-04-01 11:25:11,382][DEBUG][discovery.zen.fd] [es\_node\_4\_1]  
[master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2015-04-01 11:25:11,398][DEBUG][discovery.zen.fd] [es\_node\_4\_1]  
[node] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2015-04-01 11:25:11,400][DEBUG][transport.netty] [es\_node\_4\_2]  
using worker\_count[64], port[9301], bind\_host[null], publish\_host[null],  
compress[false], connect\_timeout[30s], connections\_per\_node[2/3/6/1/1],  
receive\_predictor[512kb-\>512kb]  
[2015-04-01 11:25:11,411][DEBUG][discovery.zen.ping.unicast] [es\_node\_4\_2]  
using initial hosts [10.60.9.4:9300, 10.60.9.4:9301, 10.60.9.5:9300,  
10.60.9.5:9301], with concurrent\_connects [10]  
[2015-04-01 11:25:11,414][DEBUG][discovery.zen] [es\_node\_4\_2]  
using ping.timeout [3s], master\_election.filter\_client [true],  
master\_election.filter\_data [false]  
[2015-04-01 11:25:11,415][DEBUG][discovery.zen.elect] [es\_node\_4\_2]  
using minimum\_master\_nodes [3]  
[2015-04-01 11:25:11,416][DEBUG][discovery.zen.fd] [es\_node\_4\_2]  
[master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2015-04-01 11:25:11,428][DEBUG][discovery.zen.fd] [es\_node\_4\_2]  
[node] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2015-04-01 11:25:11,444][DEBUG][monitor.jvm] [es\_node\_4\_1]  
enabled [true], last\_gc\_enabled [false], interval [1s], gc\_threshold  
[{old=GcThreshold{name='old', warnThreshold=10000, infoThreshold=5000,  
debugThreshold=2000}, default=GcThreshold{name='default',  
warnThreshold=10000, infoThreshold=5000, debugThreshold=2000},  
young=GcThreshold{name='young', warnThreshold=1000, infoThreshold=700,  
debugThreshold=400}}]  
[2015-04-01 11:25:11,461][DEBUG][monitor.jvm] [es\_node\_4\_2]  
enabled [true], last\_gc\_enabled [false], interval [1s], gc\_threshold  
[{old=GcThreshold{name='old', warnThreshold=10000, infoThreshold=5000,  
debugThreshold=2000}, default=GcThreshold{name='default',  
warnThreshold=10000, infoThreshold=5000, debugThreshold=2000},  
young=GcThreshold{name='young', warnThreshold=1000, infoThreshold=700,  
debugThreshold=400}}]  
[2015-04-01 11:25:11,960][DEBUG][monitor.os] [es\_node\_4\_1]  
Using probe [org.elasticsearch.monitor.os.SigarOsProbe@4d8ef117] with  
refresh\_interval [1s]  
[2015-04-01 11:25:11,971][DEBUG][monitor.process] [es\_node\_4\_1]  
Using probe [org.elasticsearch.monitor.process.SigarProcessProbe@52ed3b53]  
with refresh\_interval [1s]  
[2015-04-01 11:25:11,974][DEBUG][monitor.os] [es\_node\_4\_2]  
Using probe [org.elasticsearch.monitor.os.SigarOsProbe@4fe2fe5d] with  
refresh\_interval [1s]  
[2015-04-01 11:25:11,976][DEBUG][monitor.jvm] [es\_node\_4\_1]  
Using refresh\_interval [1s]  
[2015-04-01 11:25:11,976][DEBUG][monitor.network] [es\_node\_4\_1]  
Using probe [org.elasticsearch.monitor.network.SigarNetworkProbe@2c18b492]  
with refresh\_interval [5s]  
[2015-04-01 11:25:11,981][DEBUG][monitor.process] [es\_node\_4\_2]  
Using probe [org.elasticsearch.monitor.process.SigarProcessProbe@5230b601]  
with refresh\_interval [1s]  
[2015-04-01 11:25:11,986][DEBUG][monitor.jvm] [es\_node\_4\_2]  
Using refresh\_interval [1s]  
[2015-04-01 11:25:11,987][DEBUG][monitor.network] [es\_node\_4\_2]  
Using probe [org.elasticsearch.monitor.network.SigarNetworkProbe@44f1b25e]  
with refresh\_interval [5s]  
[2015-04-01 11:25:12,005][DEBUG][monitor.network] [es\_node\_4\_1]  
net\_info  
host [CRXJ-MONITOR-1]  
bond1 display\_name [bond1]  
address [/fe80:0:0:0:2e44:fdff:fe84:a8fe%7] [/192.168.129.4]  
mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual  
[false]  
bond0 display\_name [bond0]  
address [/fe80:0:0:0:2e44:fdff:fe84:a8fc%6] [/10.60.9.4]  
mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual  
[false]  
lo display\_name [lo]  
address [/0:0:0:0:0:0:0:1%1] [/127.0.0.1]  
mtu [16436] multicast [false] ptp [false] loopback [true] up [true] virtual  
[false]

[2015-04-01 11:25:12,005][DEBUG][monitor.network] [es\_node\_4\_2]  
net\_info  
host [CRXJ-MONITOR-1]  
bond1 display\_name [bond1]  
address [/fe80:0:0:0:2e44:fdff:fe84:a8fe%7] [/192.168.129.4]  
mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual  
[false]  
bond0 display\_name [bond0]  
address [/fe80:0:0:0:2e44:fdff:fe84:a8fc%6] [/10.60.9.4]  
mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual  
[false]  
lo display\_name [lo]  
address [/0:0:0:0:0:0:0:1%1] [/127.0.0.1]  
mtu [16436] multicast [false] ptp [false] loopback [true] up [true] virtual  
[false]

[2015-04-01 11:25:12,042][DEBUG][monitor.fs] [es\_node\_4\_1]  
Using probe [org.elasticsearch.monitor.fs.SigarFsProbe@32552379] with  
refresh\_interval [1s]  
[2015-04-01 11:25:12,047][DEBUG][monitor.fs] [es\_node\_4\_2]  
Using probe [org.elasticsearch.monitor.fs.SigarFsProbe@be389b8] with  
refresh\_interval [1s]  
[2015-04-01 11:25:12,395][DEBUG][indices.store] [es\_node\_4\_1]  
using indices.store.throttle.type [none], with  
index.store.throttle.max\_bytes\_per\_sec [20mb]  
[2015-04-01 11:25:12,404][DEBUG][indices.store] [es\_node\_4\_2]  
using indices.store.throttle.type [none], with  
index.store.throttle.max\_bytes\_per\_sec [20mb]  
[2015-04-01 11:25:12,410][DEBUG][script] [es\_node\_4\_1]  
using script cache with max\_size [500], expire [null]  
[2015-04-01 11:25:12,416][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_1] using node\_concurrent\_recoveries [2],  
node\_initial\_primaries\_recoveries [4]  
[2015-04-01 11:25:12,417][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_1] using [cluster.routing.allocation.allow\_rebalance] with  
[indices\_all\_active]  
[2015-04-01 11:25:12,417][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_1] using [cluster\_concurrent\_rebalance] with [2]  
[2015-04-01 11:25:12,419][DEBUG][script] [es\_node\_4\_2]  
using script cache with max\_size [500], expire [null]  
[2015-04-01 11:25:12,422][DEBUG][gateway.local] [es\_node\_4\_1]  
using initial\_shards [quorum], list\_timeout [30s]  
[2015-04-01 11:25:12,425][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_2] using node\_concurrent\_recoveries [2],  
node\_initial\_primaries\_recoveries [4]  
[2015-04-01 11:25:12,426][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_2] using [cluster.routing.allocation.allow\_rebalance] with  
[indices\_all\_active]  
[2015-04-01 11:25:12,427][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_2] using [cluster\_concurrent\_rebalance] with [2]  
[2015-04-01 11:25:12,431][DEBUG][gateway.local] [es\_node\_4\_2]  
using initial\_shards [quorum], list\_timeout [30s]  
[2015-04-01 11:25:12,438][DEBUG][indices.recovery] [es\_node\_4\_1]  
using max\_bytes\_per\_sec[20mb], concurrent\_streams [3], file\_chunk\_size  
[512kb], translog\_size [512kb], translog\_ops [1000], and compress [true]  
[2015-04-01 11:25:12,448][DEBUG][indices.recovery] [es\_node\_4\_2]  
using max\_bytes\_per\_sec[20mb], concurrent\_streams [3], file\_chunk\_size  
[512kb], translog\_size [512kb], translog\_ops [1000], and compress [true]  
[2015-04-01 11:25:12,583][DEBUG][http.netty] [es\_node\_4\_1]  
using max\_chunk\_size[8kb], max\_header\_size[8kb],  
max\_initial\_line\_length[4kb], max\_content\_length[100mb],  
receive\_predictor[512kb-\>512kb]  
[2015-04-01 11:25:12,589][DEBUG][indices.memory] [es\_node\_4\_1]  
using index\_buffer\_size [2.6gb], with min\_shard\_index\_buffer\_size [4mb],  
max\_shard\_index\_buffer\_size [512mb], shard\_inactive\_time [30m]  
[2015-04-01 11:25:12,590][DEBUG][indices.cache.filter] [es\_node\_4\_1]  
using [node] weighted filter cache with size [20%], actual\_size [1.5gb],  
expire [null], clean\_interval [1m]  
[2015-04-01 11:25:12,591][DEBUG][indices.fielddata.cache] [es\_node\_4\_1]  
using size [25%] [1.9gb], expire [null]  
[2015-04-01 11:25:12,594][DEBUG][http.netty] [es\_node\_4\_2]  
using max\_chunk\_size[8kb], max\_header\_size[8kb],  
max\_initial\_line\_length[4kb], max\_content\_length[100mb],  
receive\_predictor[512kb-\>512kb]  
[2015-04-01 11:25:12,601][DEBUG][indices.memory] [es\_node\_4\_2]  
using index\_buffer\_size [2.6gb], with min\_shard\_index\_buffer\_size [4mb],  
max\_shard\_index\_buffer\_size [512mb], shard\_inactive\_time [30m]  
[2015-04-01 11:25:12,602][DEBUG][indices.cache.filter] [es\_node\_4\_2]  
using [node] weighted filter cache with size [20%], actual\_size [1.5gb],  
expire [null], clean\_interval [1m]  
[2015-04-01 11:25:12,604][DEBUG][indices.fielddata.cache] [es\_node\_4\_2]  
using size [25%] [1.9gb], expire [null]  
[2015-04-01 11:25:12,609][DEBUG][gateway.local.state.meta] [es\_node\_4\_1]  
using gateway.local.auto\_import\_dangled [YES], with  
gateway.local.dangling\_timeout [2h]  
[2015-04-01 11:25:12,622][DEBUG][gateway.local.state.meta] [es\_node\_4\_2]  
using gateway.local.auto\_import\_dangled [YES], with  
gateway.local.dangling\_timeout [2h]  
[2015-04-01 11:25:12,801][DEBUG][gateway.local.state.meta] [es\_node\_4\_1]  
took 191ms to load state  
[2015-04-01 11:25:12,853][DEBUG][gateway.local.state.meta] [es\_node\_4\_2]  
took 230ms to load state  
[2015-04-01 11:25:13,112][DEBUG][gateway.local.state.shards] [es\_node\_4\_2]  
took 256ms to load started shards state  
[2015-04-01 11:25:13,118][DEBUG][bulk.udp] [es\_node\_4\_2]  
using enabled [false], host [null], port [9700-9800], bulk\_actions [1000],  
bulk\_size [5mb], flush\_interval [5s], concurrent\_requests [4]  
[2015-04-01 11:25:13,126][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_2] using node\_concurrent\_recoveries [2],  
node\_initial\_primaries\_recoveries [4]  
[2015-04-01 11:25:13,127][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_2] using [cluster.routing.allocation.allow\_rebalance] with  
[indices\_all\_active]  
[2015-04-01 11:25:13,127][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_2] using [cluster\_concurrent\_rebalance] with [2]  
[2015-04-01 11:25:13,129][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_2] using node\_concurrent\_recoveries [2],  
node\_initial\_primaries\_recoveries [4]  
[2015-04-01 11:25:13,129][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_2] using [cluster.routing.allocation.allow\_rebalance] with  
[indices\_all\_active]  
[2015-04-01 11:25:13,129][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_2] using [cluster\_concurrent\_rebalance] with [2]  
[2015-04-01 11:25:13,154][INFO][node] [es\_node\_4\_2]  
initialized  
[2015-04-01 11:25:13,154][INFO][node] [es\_node\_4\_2]  
starting ...  
[2015-04-01 11:25:13,178][DEBUG][netty.channel.socket.nio.SelectorUtil]  
Using select timeout of 500  
[2015-04-01 11:25:13,178][DEBUG][netty.channel.socket.nio.SelectorUtil]  
Epoll-bug workaround enabled = false  
[2015-04-01 11:25:13,277][DEBUG][gateway.local.state.shards] [es\_node\_4\_1]  
took 473ms to load started shards state  
[2015-04-01 11:25:13,283][DEBUG][bulk.udp] [es\_node\_4\_1]  
using enabled [false], host [null], port [9700-9800], bulk\_actions [1000],  
bulk\_size [5mb], flush\_interval [5s], concurrent\_requests [4]  
[2015-04-01 11:25:13,291][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_1] using node\_concurrent\_recoveries [2],  
node\_initial\_primaries\_recoveries [4]  
[2015-04-01 11:25:13,292][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_1] using [cluster.routing.allocation.allow\_rebalance] with  
[indices\_all\_active]  
[2015-04-01 11:25:13,292][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_1] using [cluster\_concurrent\_rebalance] with [2]  
[2015-04-01 11:25:13,294][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_1] using node\_concurrent\_recoveries [2],  
node\_initial\_primaries\_recoveries [4]  
[2015-04-01 11:25:13,294][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_1] using [cluster.routing.allocation.allow\_rebalance] with  
[indices\_all\_active]  
[2015-04-01 11:25:13,294][DEBUG][cluster.routing.allocation.decider]  
[es\_node\_4\_1] using [cluster\_concurrent\_rebalance] with [2]  
[2015-04-01 11:25:13,318][INFO][node] [es\_node\_4\_1]  
initialized  
[2015-04-01 11:25:13,318][INFO][node] [es\_node\_4\_1]  
starting ...  
[2015-04-01 11:25:13,350][DEBUG][netty.channel.socket.nio.SelectorUtil]  
Using select timeout of 500  
[2015-04-01 11:25:13,351][DEBUG][netty.channel.socket.nio.SelectorUtil]  
Epoll-bug workaround enabled = false  
[2015-04-01 11:25:13,354][DEBUG][transport.netty] [es\_node\_4\_2]  
Bound to address [/0:0:0:0:0:0:0:0:9301]  
[2015-04-01 11:25:13,355][INFO][transport] [es\_node\_4\_2]  
bound\_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish\_address  
{inet[/10.60.9.4:9301]}  
[2015-04-01 11:25:13,420][DEBUG][transport.netty] [es\_node\_4\_2]  
connected to node [[#zen\_unicast\_4#][CRXJ-MONITOR-1][inet[/10.60.9.5:9301]]]  
[2015-04-01 11:25:13,420][DEBUG][transport.netty] [es\_node\_4\_2]  
connected to node [[#zen\_unicast\_3#][CRXJ-MONITOR-1][inet[/10.60.9.5:9300]]]  
[2015-04-01 11:25:13,445][DEBUG][transport.netty] [es\_node\_4\_2]  
connected to node  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:13,543][DEBUG][transport.netty] [es\_node\_4\_1]  
Bound to address [/0:0:0:0:0:0:0:0:9300]  
[2015-04-01 11:25:13,544][INFO][transport] [es\_node\_4\_1]  
bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address  
{inet[/10.60.9.4:9300]}  
[2015-04-01 11:25:13,611][DEBUG][transport.netty] [es\_node\_4\_1]  
connected to node [[#zen\_unicast\_3#][CRXJ-MONITOR-1][inet[/10.60.9.5:9300]]]  
[2015-04-01 11:25:13,611][DEBUG][transport.netty] [es\_node\_4\_1]  
connected to node [[#zen\_unicast\_4#][CRXJ-MONITOR-1][inet[/10.60.9.5:9301]]]  
[2015-04-01 11:25:13,612][DEBUG][transport.netty] [es\_node\_4\_1]  
connected to node  
[[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:13,611][DEBUG][transport.netty] [es\_node\_4\_1]  
connected to node [[#zen\_unicast\_2#][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]]  
[2015-04-01 11:25:14,873][DEBUG][transport.netty] [es\_node\_4\_2]  
connected to node [[#zen\_unicast\_1#][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]]  
[2015-04-01 11:25:16,388][DEBUG][transport.netty] [es\_node\_4\_2]  
disconnected from [[#zen\_unicast\_4#][CRXJ-MONITOR-1][inet[/10.60.9.5:9301]]]  
[2015-04-01 11:25:16,407][DEBUG][transport.netty] [es\_node\_4\_2]  
disconnected from  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:16,409][DEBUG][transport.netty] [es\_node\_4\_2]  
disconnected from [[#zen\_unicast\_1#][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]]  
[2015-04-01 11:25:16,410][DEBUG][transport.netty] [es\_node\_4\_2]  
disconnected from [[#zen\_unicast\_3#][CRXJ-MONITOR-1][inet[/10.60.9.5:9300]]]  
[2015-04-01 11:25:16,411][DEBUG][discovery.zen] [es\_node\_4\_2]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}], master [null]  
--\> target  
[[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}], master [null]  
--\> target  
[[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}], master [null]  
[2015-04-01 11:25:16,422][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [zen-disco-join (elected\_as\_master)]: execute  
[2015-04-01 11:25:16,425][DEBUG][cluster.service] [es\_node\_4\_2]  
cluster state updated, version [1], source [zen-disco-join  
(elected\_as\_master)]  
[2015-04-01 11:25:16,427][INFO][cluster.service] [es\_node\_4\_2]  
new\_master  
[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}, reason: zen-disco-join (elected\_as\_master)  
[2015-04-01 11:25:16,436][DEBUG][transport.netty] [es\_node\_4\_2]  
connected to node  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:16,437][DEBUG][cluster.service] [es\_node\_4\_2]  
publishing cluster state version 1  
[2015-04-01 11:25:16,437][DEBUG][cluster.service] [es\_node\_4\_2]  
set local cluster state to version 1  
[2015-04-01 11:25:16,441][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: execute  
[2015-04-01 11:25:16,442][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: no change in cluster\_state  
[2015-04-01 11:25:16,442][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [zen-disco-join (elected\_as\_master)]: done applying updated  
cluster\_state (version: 1)  
[2015-04-01 11:25:16,443][INFO][discovery] [es\_node\_4\_2]  
elasticsearch\_log/CoY0ysSrTlqtSxJ6iS6wWg  
[2015-04-01 11:25:16,443][DEBUG][gateway] [es\_node\_4\_2]  
not recovering from gateway, nodes\_size (data+master) [1] \<  
recover\_after\_nodes [4]  
[2015-04-01 11:25:16,490][INFO][http] [es\_node\_4\_2]  
bound\_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish\_address  
{inet[/10.60.9.4:9201]}  
[2015-04-01 11:25:16,493][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [updating local node id]: execute  
[2015-04-01 11:25:16,493][DEBUG][cluster.service] [es\_node\_4\_2]  
cluster state updated, version [2], source [updating local node id]  
[2015-04-01 11:25:16,493][DEBUG][cluster.service] [es\_node\_4\_2]  
publishing cluster state version 2  
[2015-04-01 11:25:16,496][DEBUG][cluster.service] [es\_node\_4\_2]  
set local cluster state to version 2  
[2015-04-01 11:25:16,497][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: execute  
[2015-04-01 11:25:16,497][DEBUG][gateway] [es\_node\_4\_2]  
not recovering from gateway, nodes\_size (data+master) [1] \<  
recover\_after\_nodes [4]  
[2015-04-01 11:25:16,497][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: no change in cluster\_state  
[2015-04-01 11:25:16,497][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [updating local node id]: done applying updated cluster\_state  
(version: 2)  
[2015-04-01 11:25:16,497][INFO][node] [es\_node\_4\_2]  
started  
[2015-04-01 11:25:16,520][DEBUG][transport.netty] [es\_node\_4\_2]  
connected to node  
[[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:16,531][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [zen-disco-receive(join from  
node[[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}])]: execute  
[2015-04-01 11:25:16,532][DEBUG][cluster.service] [es\_node\_4\_2]  
cluster state updated, version [3], source [zen-disco-receive(join from  
node[[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}])]  
[2015-04-01 11:25:16,532][INFO][cluster.service] [es\_node\_4\_2]  
added  
{[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2},}, reason: zen-disco-receive(join from  
node[[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}])  
[2015-04-01 11:25:16,532][DEBUG][cluster.service] [es\_node\_4\_2]  
publishing cluster state version 3  
[2015-04-01 11:25:16,570][DEBUG][cluster.service] [es\_node\_4\_2]  
set local cluster state to version 3  
[2015-04-01 11:25:16,570][DEBUG][cluster] [es\_node\_4\_2]  
data node was added, retrieving new cluster info  
[2015-04-01 11:25:16,572][DEBUG][gateway] [es\_node\_4\_2]  
not recovering from gateway, nodes\_size (data+master) [2] \<  
recover\_after\_nodes [4]  
[2015-04-01 11:25:16,572][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: execute  
[2015-04-01 11:25:16,572][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [zen-disco-receive(join from  
node[[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}])]: done applying updated cluster\_state  
(version: 3)  
[2015-04-01 11:25:16,572][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: no change in cluster\_state  
[2015-04-01 11:25:16,581][DEBUG][transport.netty] [es\_node\_4\_1]  
disconnected from [[#zen\_unicast\_4#][CRXJ-MONITOR-1][inet[/10.60.9.5:9301]]]  
[2015-04-01 11:25:16,591][ERROR][cluster] [es\_node\_4\_2]  
Failed to execute IndicesStatsAction for ClusterInfoUpdateJob  
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:  
[SERVICE\_UNAVAILABLE/1/state not recovered / initialized];  
at  
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)  
at  
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:89)  
at  
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:53)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:124)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:75)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46)  
at  
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)  
at  
org.elasticsearch.cluster.InternalClusterInfoService$ClusterInfoUpdateJob.run(InternalClusterInfoService.java:293)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)  
at java.lang.Thread.run(Thread.java:662)  
[2015-04-01 11:25:16,600][DEBUG][transport.netty] [es\_node\_4\_1]  
disconnected from  
[[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:16,601][DEBUG][transport.netty] [es\_node\_4\_1]  
disconnected from [[#zen\_unicast\_3#][CRXJ-MONITOR-1][inet[/10.60.9.5:9300]]]  
[2015-04-01 11:25:16,602][DEBUG][transport.netty] [es\_node\_4\_1]  
disconnected from [[#zen\_unicast\_2#][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]]  
[2015-04-01 11:25:16,603][DEBUG][discovery.zen] [es\_node\_4\_1]  
filtered ping responses: (filter\_client[true], filter\_data[false])  
--\> target  
[[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}], master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
--\> target  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}], master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
--\> target  
[[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}], master [null]  
[2015-04-01 11:25:16,615][DEBUG][transport.netty] [es\_node\_4\_1]  
connected to node  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:16,628][DEBUG][transport.netty] [es\_node\_4\_2]  
connected to node  
[[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:16,636][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [zen-disco-receive(join from  
node[[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]: execute  
[2015-04-01 11:25:16,636][DEBUG][cluster.service] [es\_node\_4\_2]  
cluster state updated, version [4], source [zen-disco-receive(join from  
node[[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]  
[2015-04-01 11:25:16,636][INFO][cluster.service] [es\_node\_4\_2]  
added  
{[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2},}, reason: zen-disco-receive(join from  
node[[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])  
[2015-04-01 11:25:16,637][DEBUG][cluster.service] [es\_node\_4\_2]  
publishing cluster state version 4  
[2015-04-01 11:25:16,638][DEBUG][discovery.zen.fd] [es\_node\_4\_1]  
[master] starting fault detection against master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}], reason [initial\_join]  
[2015-04-01 11:25:16,646][DEBUG][discovery.zen.publish] [es\_node\_4\_1]  
received cluster state version 4  
[2015-04-01 11:25:16,648][DEBUG][discovery.zen] [es\_node\_4\_1]  
received cluster state from  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}] which is also master but with cluster name  
[Cluster [elasticsearch\_log]]  
[2015-04-01 11:25:16,656][DEBUG][cluster.service] [es\_node\_4\_1]  
processing [zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]: execute  
[2015-04-01 11:25:16,658][DEBUG][cluster.service] [es\_node\_4\_1]  
got first state from fresh master [CoY0ysSrTlqtSxJ6iS6wWg]  
[2015-04-01 11:25:16,658][DEBUG][cluster.service] [es\_node\_4\_1]  
cluster state updated, version [4], source [zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]  
[2015-04-01 11:25:16,660][INFO][cluster.service] [es\_node\_4\_1]  
detected\_master  
[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}, added  
{[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2},[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2},}, reason: zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])  
[2015-04-01 11:25:16,671][DEBUG][transport.netty] [es\_node\_4\_1]  
connected to node  
[[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:16,675][DEBUG][transport.netty] [es\_node\_4\_1]  
connected to node  
[[es\_node\_5\_2][UhG2Mtf4SVOgAZLpm7GNuQ][CRXJ-MONITOR-2][inet[/10.60.9.5:9301]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:16,675][DEBUG][cluster.service] [es\_node\_4\_1]  
set local cluster state to version 4  
[2015-04-01 11:25:16,676][DEBUG][cluster.service] [es\_node\_4\_1]  
processing [zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]: done applying updated cluster\_state  
(version: 4)  
[2015-04-01 11:25:16,676][INFO][discovery] [es\_node\_4\_1]  
elasticsearch\_log/w58vSDleSS2geuwMrChlxg  
[2015-04-01 11:25:16,677][DEBUG][cluster.service] [es\_node\_4\_2]  
set local cluster state to version 4  
[2015-04-01 11:25:16,678][DEBUG][cluster] [es\_node\_4\_2]  
data node was added, retrieving new cluster info  
[2015-04-01 11:25:16,678][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: execute  
[2015-04-01 11:25:16,678][DEBUG][gateway] [es\_node\_4\_2]  
not recovering from gateway, nodes\_size (data+master) [3] \<  
recover\_after\_nodes [4]  
[2015-04-01 11:25:16,678][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: no change in cluster\_state  
[2015-04-01 11:25:16,678][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [zen-disco-receive(join from  
node[[es\_node\_4\_1][w58vSDleSS2geuwMrChlxg][CRXJ-MONITOR-1][inet[/10.60.9.4:9300]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]: done applying updated cluster\_state  
(version: 4)  
[2015-04-01 11:25:16,679][ERROR][cluster] [es\_node\_4\_2]  
Failed to execute IndicesStatsAction for ClusterInfoUpdateJob  
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:  
[SERVICE\_UNAVAILABLE/1/state not recovered / initialized];  
at  
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)  
at  
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:89)  
at  
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:53)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:124)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:75)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46)  
at  
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)  
at  
org.elasticsearch.cluster.InternalClusterInfoService$ClusterInfoUpdateJob.run(InternalClusterInfoService.java:293)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)  
at java.lang.Thread.run(Thread.java:662)  
[2015-04-01 11:25:16,723][INFO][http] [es\_node\_4\_1]  
bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address  
{inet[/10.60.9.4:9200]}  
[2015-04-01 11:25:16,725][DEBUG][cluster.service] [es\_node\_4\_1]  
processing [updating local node id]: execute  
[2015-04-01 11:25:16,726][DEBUG][cluster.service] [es\_node\_4\_1]  
cluster state updated, version [4], source [updating local node id]  
[2015-04-01 11:25:16,726][DEBUG][cluster.service] [es\_node\_4\_1]  
set local cluster state to version 4  
[2015-04-01 11:25:16,726][DEBUG][cluster.service] [es\_node\_4\_1]  
processing [updating local node id]: done applying updated cluster\_state  
(version: 4)  
[2015-04-01 11:25:16,726][INFO][node] [es\_node\_4\_1]  
started  
[2015-04-01 11:25:18,312][DEBUG][transport.netty] [es\_node\_4\_2]  
connected to node  
[[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:18,318][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [zen-disco-receive(join from  
node[[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}])]: execute  
[2015-04-01 11:25:18,319][DEBUG][cluster.service] [es\_node\_4\_2]  
cluster state updated, version [5], source [zen-disco-receive(join from  
node[[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}])]  
[2015-04-01 11:25:18,319][INFO][cluster.service] [es\_node\_4\_2]  
added  
{[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2},}, reason: zen-disco-receive(join from  
node[[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}])  
[2015-04-01 11:25:18,320][DEBUG][cluster.service] [es\_node\_4\_2]  
publishing cluster state version 5  
[2015-04-01 11:25:18,322][DEBUG][discovery.zen.publish] [es\_node\_4\_1]  
received cluster state version 5  
[2015-04-01 11:25:18,323][DEBUG][discovery.zen] [es\_node\_4\_1]  
received cluster state from  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}] which is also master but with cluster name  
[Cluster [elasticsearch\_log]]  
[2015-04-01 11:25:18,323][DEBUG][cluster.service] [es\_node\_4\_1]  
processing [zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]: execute  
[2015-04-01 11:25:18,324][DEBUG][cluster.service] [es\_node\_4\_1]  
cluster state updated, version [5], source [zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]  
[2015-04-01 11:25:18,324][INFO][cluster.service] [es\_node\_4\_1]  
added  
{[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2},}, reason: zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])  
[2015-04-01 11:25:18,329][DEBUG][transport.netty] [es\_node\_4\_1]  
connected to node  
[[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}]  
[2015-04-01 11:25:18,329][DEBUG][cluster.service] [es\_node\_4\_1]  
set local cluster state to version 5  
[2015-04-01 11:25:18,329][DEBUG][cluster.service] [es\_node\_4\_1]  
processing [zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]: done applying updated cluster\_state  
(version: 5)  
[2015-04-01 11:25:18,356][DEBUG][cluster.service] [es\_node\_4\_2]  
set local cluster state to version 5  
[2015-04-01 11:25:18,356][DEBUG][cluster] [es\_node\_4\_2]  
data node was added, retrieving new cluster info  
[2015-04-01 11:25:18,357][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: execute  
[2015-04-01 11:25:18,357][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: no change in cluster\_state  
[2015-04-01 11:25:18,359][ERROR][cluster] [es\_node\_4\_2]  
Failed to execute IndicesStatsAction for ClusterInfoUpdateJob  
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:  
[SERVICE\_UNAVAILABLE/1/state not recovered / initialized];  
at  
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)  
at  
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:89)  
at  
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:53)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:124)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:75)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46)  
at  
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)  
at  
org.elasticsearch.cluster.InternalClusterInfoService$ClusterInfoUpdateJob.run(InternalClusterInfoService.java:293)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)  
at java.lang.Thread.run(Thread.java:662)  
[2015-04-01 11:25:18,360][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [zen-disco-receive(join from  
node[[es\_node\_5\_1][nwkFlS5uQGSbdYjzuWKUrg][CRXJ-MONITOR-2][inet[/10.60.9.5:9300]]{rack\_id=rack\_node\_5,  
max\_local\_storage\_nodes=2}])]: done applying updated cluster\_state  
(version: 5)  
[2015-04-01 11:25:18,650][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [local-gateway-elected-state]: execute  
[2015-04-01 11:25:18,676][DEBUG][cluster.service] [es\_node\_4\_2]  
cluster state updated, version [6], source [local-gateway-elected-state]  
[2015-04-01 11:25:18,677][DEBUG][cluster.service] [es\_node\_4\_2]  
publishing cluster state version 6  
[2015-04-01 11:25:18,713][DEBUG][discovery.zen.publish] [es\_node\_4\_1]  
received cluster state version 6  
[2015-04-01 11:25:18,713][DEBUG][discovery.zen] [es\_node\_4\_1]  
received cluster state from  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}] which is also master but with cluster name  
[Cluster [elasticsearch\_log]]  
[2015-04-01 11:25:18,714][DEBUG][cluster.service] [es\_node\_4\_1]  
processing [zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]: execute  
[2015-04-01 11:25:18,718][DEBUG][cluster.service] [es\_node\_4\_1]  
cluster state updated, version [6], source [zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]  
[2015-04-01 11:25:18,718][DEBUG][cluster.service] [es\_node\_4\_1]  
set local cluster state to version 6  
[2015-04-01 11:25:19,247][DEBUG][cluster.service] [es\_node\_4\_1]  
processing [zen-disco-receive(from master  
[[es\_node\_4\_2][CoY0ysSrTlqtSxJ6iS6wWg][CRXJ-MONITOR-1][inet[/10.60.9.4:9301]]{rack\_id=rack\_node\_4,  
max\_local\_storage\_nodes=2}])]: done applying updated cluster\_state  
(version: 6)  
[2015-04-01 11:25:19,302][DEBUG][cluster.service] [es\_node\_4\_2]  
set local cluster state to version 6  
[2015-04-01 11:25:19,311][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: execute  
[2015-04-01 11:25:19,312][DEBUG][river.cluster] [es\_node\_4\_2]  
processing [reroute\_rivers\_node\_changed]: no change in cluster\_state  
[2015-04-01 11:25:19,579][INFO][gateway] [es\_node\_4\_2]  
recovered [84] indices into cluster\_state  
[2015-04-01 11:25:19,579][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [local-gateway-elected-state]: done applying updated  
cluster\_state (version: 6)  
[2015-04-01 11:25:26,440][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [routing-table-updater]: execute  
[2015-04-01 11:25:26,441][DEBUG][cluster.service] [es\_node\_4\_2]  
processing [routing-table-updater]: no change in cluster\_state

elasticsearch.yml  
index.merge.policy.max\_merged\_segment: 1gb  
index.merge.policy.segments\_per\_tier: 4  
index.merge.policy.max\_merge\_at\_once: 4  
index.merge.policy.max\_merge\_at\_once\_explicit: 4  
index.merge.scheduler.max\_thread\_count: 1  
indices.memory.index\_buffer\_size: 33%  
indices.store.throttle.type: none  
threadpool.merge.type: fixed  
threadpool.merge.size: 4  
threadpool.merge.queue\_size: 32  
###bulk线程池类型fixed-固定  
threadpool.bulk.type: fixed  
####bulk线程最大值  
threadpool.bulk.size: 20  
####bulk队列最大值  
threadpool.bulk.queue\_size: 32  
bootstrap.mlockall: true  
node.max\_local\_storage\_nodes: 2  
cluster.name: elasticsearch\_log

http.port: 9201  
transport.tcp.port: 9301  
node.name: es\_node\_4\_2  
path.data: /home/nat/esdata.d  
node.rack\_id: rack\_node\_4  
cluster.routing.allocation.awareness.attributes: rack\_id  
#######################################  
###zen 避免分裂  
discovery.zen.minimum\_master\_nodes: 3  
discovery.zen.ping.multicast.enabled: false  
discovery.zen.ping.unicast.hosts:  
["10.60.9.4:9300","10.60.9.4:9301","10.60.9.5:9300","10.60.9.5:9301"]  
###字段数据缓存  
indices.fielddata.cache.size: 25%  
###shards 分配  
cluster.routing.allocation.disk.threshold\_enabled: true  
cluster.routing.allocation.disk.watermark.low: 97  
cluster.routing.allocation.disk.watermark.high: 99  
###开始恢复的最小节点数  
gateway.recover\_after\_nodes: 4  
###索引存储类型  
index.store.type: niofs  
###数据压缩  
index.store.compress.stored: true  
index.store.compress.tv: true  
######################################

I tried various ways:open 9300-9400 port (tcp), close firewall, recovery  
hosts, etc.But they are useless.

Anybody have any suggestions about this?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fd4ae959-671b-445c-bed8-64e5599b6245%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fd4ae959-671b-445c-bed8-64e5599b6245%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:22am UTC](https://discuss.elastic.co/t/a-problem-clusterblockexception-blocked-by-service-unavailable-1-state-not-recovered-initialized/23027/2 "2017-07-06T00:22:28Z")

</div>