# \[ESv5.6\] Cluster and shards problems
**URL:** https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389
**Category:** Elasticsearch
**Created:** [December 19, 2017, 9:22am UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389 "2017-12-19T09:22:09Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@alias](https://discuss.elastic.co/u/alias)
#### Post date: [December 19, 2017, 9:22am UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/1 "2017-12-19T09:22:10Z")
Hi all,
I've cluster and shards problem on my ES cluster in v5.6.
Node01 :
```auto
# cat /etc/elasticsearch/elasticsearch.yml | grep -v "^#"
cluster.name: escluster
node.name: node-01
node.master: true
node.data: false
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: ["127.0.0.1","10.0.0.1"]
http.port: 9200
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]
discovery.zen.minimum_master_nodes: 2
action.auto_create_index: false
# cat /etc/sysconfig/elasticsearch | grep -v "^#"
LOG_DIR=/var/log/elasticsearch
ES_JAVA_OPTS="-Xms2g -Xmx2g"
ES_STARTUP_SLEEP_TIME=5
```
Node02 :
```auto
# cat /etc/elasticsearch/elasticsearch.yml | grep -v "^#"
cluster.name: escluster
node.name: node-02
node.master: false
node.data: true
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: ["127.0.0.1","10.0.0.2"]
http.port: 9200
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]
discovery.zen.minimum_master_nodes: 2
action.auto_create_index: false
# cat /etc/sysconfig/elasticsearch | grep -v "^#"
LOG_DIR=/var/log/elasticsearch
ES_JAVA_OPTS="-Xms2g -Xmx2g"
ES_STARTUP_SLEEP_TIME=5
```
Node03 :
```auto
# cat /etc/elasticsearch/elasticsearch.yml | grep -v "^#"
cluster.name: escluster
node.name: node-03
node.master: false
node.data: true
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: ["127.0.0.1","10.0.0.3"]
http.port: 9200
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]
discovery.zen.minimum_master_nodes: 2
action.auto_create_index: false
# cat /etc/sysconfig/elasticsearch | grep -v "^#"
LOG_DIR=/var/log/elasticsearch
ES_JAVA_OPTS="-Xms2g -Xmx2g"
ES_STARTUP_SLEEP_TIME=5
```
For each node, when I restart the service, it's up for \< 1minute and pass to failed state.
In logs :
```auto
unknown setting [discovery.zen.ping.multicast.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
```
I comment this line in the configuration file and retry : the service is up. If we check the logs :
```auto
[WARN][o.e.d.z.ZenDiscovery] [node-01] not enough master nodes discovered during pinging (found [[Candidate{node={node-01}{-ZFUWniuRaeJOoHVnKw6fQ}{sgMbuNMIQHSgQDMuGflU6w}{10.0.0.1}{10.0.0.1:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
```
We change the configuration to have 2 masters :
Node01 :
```auto
# cat /etc/elasticsearch/elasticsearch.yml | grep -v "^#"
cluster.name: escluster
node.name: node-01
node.master: true
node.data: false
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: ["127.0.0.1","10.0.0.1"]
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]
discovery.zen.minimum_master_nodes: 2
action.auto_create_index: false
```
Node02 :
```auto
# cat /etc/elasticsearch/elasticsearch.yml | grep -v "^#"
cluster.name: escluster
node.name: node-02
node.master: true
node.data: true
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: ["127.0.0.1","10.0.0.2"]
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]
discovery.zen.minimum_master_nodes: 2
action.auto_create_index: false
```
Node03 :
```auto
# cat /etc/elasticsearch/elasticsearch.yml | grep -v "^#"
cluster.name: escluster
node.name: node-03
node.master: false
node.data: true
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: ["127.0.0.1","10.0.0.3"]
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]
discovery.zen.minimum_master_nodes: 2
action.auto_create_index: false
```
We restart the ES service on 3 nodes : we have no errors and no warning.
Now, I check the index and shard status :
```auto
# curl -XGET 'localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "escluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 2,
"active_primary_shards" : 8,
"active_shards" : 16,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 8,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 66.66666666666666
}
```
The cluster status is red ...
I check the state :
```auto
# curl -XGET http://127.0.0.1:9200/_cluster/state?pretty
{
"cluster_name" : "escluster",
"version" : 16,
"state_uuid" : "KeEWQGNbROabuOLFSN_xLw",
"master_node" : "-ZFUWniuRaeJOoHVnKw6fQ",
"blocks" : { },
"nodes" : {
[3 nodes in cluster]
},
[...]
"index_0" : {
"shards" : {
"3" : [
{
"state" : "UNASSIGNED",
"primary" : true,
"node" : null,
"relocating_node" : null,
"shard" : 3,
"index" : "index_0",
"recovery_source" : {
"type" : "EXISTING_STORE"
},
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2017-12-19T09:34:24.810Z",
"delayed" : false,
"allocation_status" : "no_valid_shard_copy"
}
},
{
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 3,
"index" : "index_0",
"recovery_source" : {
"type" : "PEER"
},
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2017-12-19T09:34:24.810Z",
"delayed" : false,
"allocation_status" : "no_attempt"
```
(I cut the output for post limit)
We see several shards in UNASSIGNED state.
Someone could help me to resolv the red status and unassigned shards ?
Thanks !!
---
### Author:  [@Fram\_Souza](https://discuss.elastic.co/u/Fram_Souza)
#### Post date: [December 19, 2017, 11:49am UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/2 "2017-12-19T11:49:46Z")
Hi @alias
It is recommended that you have at least 3 eligible masters, in which case "discovery.zen.minimum\_master\_nodes" must exist only on machines that are eligible, you can remove it from node03
---
### Author:  [@alias](https://discuss.elastic.co/u/alias)
#### Post date: [December 19, 2017, 3:47pm UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/3 "2017-12-19T15:47:39Z")
Hi @Fram_Souza
I delete this parameter on node03.
I've the same errors : Cluster red and UNASSIGNED shards.
Any idea ?
Thanks
---
### Author:  [@Fram\_Souza](https://discuss.elastic.co/u/Fram_Souza)
#### Post date: [December 19, 2017, 3:50pm UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/4 "2017-12-19T15:50:14Z")
@alias
You have already run the \_cluster/reroute ?
---
### Author:  [@zqc0512](https://discuss.elastic.co/u/zqc0512)
#### Post date: [December 20, 2017, 6:59am UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/5 "2017-12-20T06:59:02Z")
date nodes is only 2 you set replication is 2? so that you need 3 data nodes.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 20, 2017, 7:42am UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/6 "2017-12-20T07:42:05Z")
> [@Fram\_Souza](#):
>
> "discovery.zen.minimum\_master\_nodes" must exist only on machines that are eligible, you can remove it from node03
You should always define minimum masters on **all nodes in a cluster** , not just the masters.
> [@alias](#):
>
> I've the same errors : Cluster red and UNASSIGNED shards.
>
> Any idea ?
What does the output from `_cat/shards` look like
---
### Author:  [@alias](https://discuss.elastic.co/u/alias)
#### Post date: [January 5, 2018, 9:30am UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/7 "2018-01-05T09:30:49Z")
After differents try, all shards are started and 0 are unassigned. The cluster is green.
I've delete all data and indices, readd the minimum\_master\_nodes parameter.
Thanks you !
---
### Author:  [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)
#### Post date: [January 5, 2018, 9:35am UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/8 "2018-01-05T09:35:16Z")
@alias Please check the Elasticsearch version on all the nodes. I had a similar issue where shards were not getting allocated because I had made a cluster out of nodes have different ES versions
---
### Author:  [@alias](https://discuss.elastic.co/u/alias)
#### Post date: [January 5, 2018, 9:52am UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/9 "2018-01-05T09:52:04Z")
@Abhilash_Bolla thanks, but it's one of the first check and all version are OK (ES, and Lucene)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 2, 2018, 9:52am UTC](https://discuss.elastic.co/t/esv5-6-cluster-and-shards-problems/112389/10 "2018-02-02T09:52:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.