Rolling upgrade from 2.1.0 to 2.1.1 failed. please help to identify the probelm

Elastic Stack Elasticsearch
1

stop one of them, upgrade it and restart it. below is discovery log and it seems the server can connected to all of other nodes.
[2015-12-25 10:57:00,313][TRACE][discovery.zen.ping.unicast] [192.168.200.190] [119] received response from {#zen_unicast_3#}{192.168.200.196}{192.168.200.196:9390}: [ping_response{node [{192.168.200.190}{QDVnI6x7RQ2u3T3zBsKQgw}{192.168.200.190}{192.168.200.190:9390}{max_local_storage_nodes=20, master=false}], id[351], master [null], hasJoinedOnce [false], cluster_name[21test]}, ping_response{node [{192.168.200.190}{QDVnI6x7RQ2u3T3zBsKQgw}{192.168.200.190}{192.168.200.190:9390}{max_local_storage_nodes=20, master=false}], id[352], master [null], hasJoinedOnce [false], cluster_name[21test]}, ping_response{node [{192.168.200.190}{QDVnI6x7RQ2u3T3zBsKQgw}{192.168.200.190}{192.168.200.190:9390}{max_local_storage_nodes=20, master=false}], id[353], master [null], hasJoinedOnce [false], cluster_name[21test]}, ping_response{node [{192.168.200.190}{QDVnI6x7RQ2u3T3zBsKQgw}{192.168.200.190}{192.168.200.190:9390}{max_local_storage_nodes=20, master=false}], id[354], master [null], hasJoinedOnce [false], cluster_name[21test]}, ping_response{node [{192.168.200.190}{QDVnI6x7RQ2u3T3zBsKQgw}{192.168.200.190}{192.168.200.190:9390}{max_local_storage_nodes=20, master=false}], id[355], master [null], hasJoinedOnce [false], cluster_name[21test]}, ping_response{node [{192.168.200.190}{QDVnI6x7RQ2u3T3zBsKQgw}{192.168.200.190}{192.168.200.190:9390}{max_local_storage_nodes=20, master=false}], id[356], master [null], hasJoinedOnce [false], cluster_name[21test]}, ping_response{node [{192.168.200.196}{FzxGsQclQ22izuwoi-wTpQ}{192.168.200.196}{192.168.200.196:9390}], id[1081], master [{192.168.200.196}{FzxGsQclQ22izuwoi-wTpQ}{192.168.200.196}{192.168.200.196:9390}], hasJoinedOnce [true], cluster_name[21test]}]

but it keep logging: [DEBUG][action.admin.indices.create] [192.168.200.190] no known master node, scheduling a retry

2

cluster toplogy: 4 nodes, three of them are master&data node, the other is data node.
the config is below:
cluster.name: 21test
node.name: 192.168.200.190
node.master: false
node.data: true
path.data: /export/Data/elasticsearch/es219290
path.logs: /export/Logs/elasticsearch/es219290
path.plugins: /export/plugins/elasticsearch/es219290
bootstrap.mlockall: true
network.host: 192.168.200.190
http.port: 9290
transport.tcp.port: 9390
discovery.zen.ping.unicast.hosts: ["192.168.200.191:9390","192.168.200.192:9390","192.168.200.196:9390"]
discovery.zen.minimum_master_nodes: 3
indices.memory.index_buffer_size: 20%
script.engine.groovy.inline.aggs: on
index.translog.sync_interval: 30s
index.translog.durability: async
index.refresh_interval: 180s
index.max_merged_segment: 1g
node.max_local_storage_nodes: 20

3

If you have 3 master eligible nodes in the cluster, minimum_master_nodes should be set to 2, not 3. 3 would however be the correct setting if all 4 nodes were master eligible.