# Unresponsive Cluster

**URL:** <https://discuss.elastic.co/t/unresponsive-cluster/65070>\
**Category:** Elasticsearch\
**Created:** [November 4, 2016, 11:09pm UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070 "2016-11-04T23:09:35Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![fvegaucr](https://avatars.discourse-cdn.com/v4/letter/f/7c8e57/32.png) [@fvegaucr](https://discuss.elastic.co/u/fvegaucr)\
**Post date:** [November 4, 2016, 11:09pm UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/1 "2016-11-04T23:09:35Z")

</div>

Im having issues with my cluster.The cluster has 3 master and 15 nodes 13 are data nodes and 2 are edge nodes. The cluster was working before, and I did a change in one of the templates that we use.  
The original config looks like:

> {  
> "template" : "servers-_",  
> "settings" : {  
> "index.refresh\_interval" : "5s",  
> "index.routing.allocation.require.node\_type": "hot",  
> "analysis" : {  
> "analyzer" : {  
> "default" : {  
> "type" : "standard",  
> "stopwords" : "none"  
> }  
> }  
> }  
> },  
> "mappings" : {  
> "default" : {  
> "\_all" : {"enabled" : true},  
> "dynamic\_templates" : [  
> {  
> "string\_fields" : {  
> "match" : "_",  
> "match\_mapping\_type" : "string",  
> "mapping" : {  
> "type" : "multi\_field",  
> "fields" : {  
> "{name}" : {"type": "string", "index" : "analyzed", "omit\_norms" : true, "index\_options" : "docs"}  
> }  
> }  
> }  
> } ],  
> "properties" : {  
> "@version": { "type": "string", "index": "not\_analyzed" },  
> "geoip" : {  
> "type" : "object",  
> "dynamic": true,  
> "properties" : {  
> "ip" : { "type": "ip" },  
> "location" : { "type" : "geo\_point" }  
> }  
> },  
> "tags": { "type": "string", "index": "not\_analyzed" },  
> "pid": { "type": "long", "index": "not\_analyzed" },  
> "priority": { "type": "integer", "index": "not\_analyzed" },  
> "severity": { "type": "integer", "index": "not\_analyzed" },  
> "facility": { "type": "integer", "index": "not\_analyzed" },  
> "syslog\_severity\_code": { "type": "integer", "index": "not\_analyzed" },  
> "syslog\_facility\_code": { "type": "integer", "index": "not\_analyzed" }  
> }  
> }  
> }  
> }

My changes:

> {  
> "template" : "servers-_",  
> "settings" : {  
> "index.refresh\_interval" : "5s",  
> "index.routing.allocation.require.node\_type": "hot",  
> "analysis" : {  
> "analyzer" : {  
> "default" : {  
> "type" : "standard",  
> "stopwords" : "none"  
> }  
> }  
> }  
> },  
> "mappings" : {  
> "default" : {  
> "\_all" : {"enabled" : true},  
> "dynamic\_templates" : [  
> {  
> "beat" : {  
> "path\_match" : "beat._",  
> "match\_mapping\_type" : "string",  
> "mapping" : {  
> "type" : "string",  
> "index" : "not\_analyzed",  
> "omit\_norms" : true,  
> "index\_options" : "offsets"  
> }  
> }  
> },  
> {  
> "string\_fields" : {  
> "match" : "\*",  
> "match\_mapping\_type" : "string",  
> "mapping" : {  
> "type" : "multi\_field",  
> "fields" : {  
> "{name}" : {"type": "string", "index" : "analyzed", "omit\_norms" : true, "index\_options" : "docs"}  
> }  
> }  
> }  
> } ],  
> "properties" : {  
> "@version": { "type": "string", "index": "not\_analyzed" },  
> "geoip" : {  
> "type" : "object",  
> "dynamic": true,  
> "properties" : {  
> "ip" : { "type": "ip" },  
> "location" : { "type" : "geo\_point" }  
> }  
> },  
> "tags": { "type": "string", "index": "not\_analyzed" },  
> "pid": { "type": "long", "index": "not\_analyzed" },  
> "priority": { "type": "integer", "index": "not\_analyzed" },  
> "severity": { "type": "integer", "index": "not\_analyzed" },  
> "facility": { "type": "integer", "index": "not\_analyzed" },  
> "syslog\_severity\_code": { "type": "integer", "index": "not\_analyzed" },  
> "syslog\_facility\_code": { "type": "integer", "index": "not\_analyzed" }  
> }  
> }  
> }  
> }

Since then the cluster has being degraded to the point that I had to restarted a few times but never recovers completely.Is there a way I can restore my cluster, I cant delete the new indices or perform any kind of action due to I always get the following error:

> "error": {  
> "root\_cause": [  
> {  
> "type": "master\_not\_discovered\_exception",  
> "reason": null  
> }  
> ],  
> "type": "master\_not\_discovered\_exception",  
> "reason": null  
> },  
> "status": 503

The cluster seem to be crazy busy trying to recover indices or something else. Any suggestion of what I can do we are talking of about 1000+ indices some of them are big as 160GB

---

<div class="post-metadata">

**Author:** ![fvegaucr](https://avatars.discourse-cdn.com/v4/letter/f/7c8e57/32.png) [@fvegaucr](https://discuss.elastic.co/u/fvegaucr)\
**Post date:** [November 4, 2016, 11:11pm UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/2 "2016-11-04T23:11:01Z")

</div>

Here some sample of the logs:

> [2016-11-04 23:09:28,984][DEBUG][action.admin.indices.mapping.put] [[dwh-head003.sing1.com](http://dwh-head003.sing1.com)] failed to put mappings on indices [[servers-forecaster\_aggregator-2016.11.04]], type [servers]  
> ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [servers]) within 30s]  
> at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:349)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> at java.lang.Thread.run(Thread.java:744)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 5, 2016, 5:08am UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/3 "2016-11-05T05:08:10Z")

</div>

What version are you on?  
Do you have Monitoring in place?

---

<div class="post-metadata">

**Author:** ![fvegaucr](https://avatars.discourse-cdn.com/v4/letter/f/7c8e57/32.png) [@fvegaucr](https://discuss.elastic.co/u/fvegaucr)\
**Post date:** [November 5, 2016, 6:30am UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/4 "2016-11-05T06:30:43Z")

</div>

> name": "[edge001.sing1.com](http://edge001.sing1.com)",  
> "cluster\_name": "prodexpansion",  
> "version": {  
> "number": "2.3.1",  
> "build\_hash": "bd980929010aef404e7cb0843e61d0665269fc39",  
> "build\_timestamp": "2016-04-04T12:25:05Z",  
> "build\_snapshot": false,  
> "lucene\_version": "5.5.0"  
> },  
> "tagline": "You Know, for Search"  
> }

---

<div class="post-metadata">

**Author:** ![fvegaucr](https://avatars.discourse-cdn.com/v4/letter/f/7c8e57/32.png) [@fvegaucr](https://discuss.elastic.co/u/fvegaucr)\
**Post date:** [November 7, 2016, 7:31pm UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/5 "2016-11-07T19:31:49Z")

</div>

We are using version 2.3.1

---

<div class="post-metadata">

**Author:** ![fvegaucr](https://avatars.discourse-cdn.com/v4/letter/f/7c8e57/32.png) [@fvegaucr](https://discuss.elastic.co/u/fvegaucr)\
**Post date:** [November 7, 2016, 11:47pm UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/6 "2016-11-07T23:47:25Z")

</div>

I have also another question:

Why is that when I curl -XGET [http://head001.sing1.com:9200/\_cluster/settings](http://head001.sing1.com:9200/_cluster/settings), I get

```
{
"persistent": {},
"transient": {}
}

```

If the config file show:

```
--
action:
  disable_delete_all_indices: 1
bootstrap:
  mlockall: true
cluster:
  info:
    update:
      interval: 1m
  name: prod_sing
  routing:
    allocation:
      cluster_concurrent_rebalance: 25
      disk:
        threshold_enabled: true
        watermark:
          high: 90%
          low: 80%
discovery:
  zen:
    minimum_master_nodes: 2
    ping:
      multicast:
        enabled: false
      unicast:
        hosts:
             - kdb012.sing1.com
             - kdb013.sing1.com
             - kdb014.sing1.com
             - kdb015.sing1.com
             - kdb016.sing1.com
             - kdb017.sing1.com
             - kdb018.sing1.com
             - kdb019.sing1.com
             - edge002.sing1.com
gateway:
  recover_after_nodes: 2
http:
  cors:
    allow-origin: /.*/
    enabled: true
  enabled: true
  port: 9200
index:
  fielddata:
    cache:
      size: 40%
  number_of_replicas: 1
  number_of_shards: 15
  translog:
    flush_threshold_ops: 50000
indices:
  memory:
    index_buffer_size: 50%
network:
  bind_host: 0.0.0.0
  publish_host: 10.0.0.1
node:
  data: true
  master: false
  name: kdb006.sing1.com-hot
  node_type: hot
path:
  data: /data/hot
threadpool:
  index:
    queue_size: 200
    size: 24
    type: fixed
  management:
    size: 20
  search:
    queue_size: 100
    size: 8
    type: fixed
transport:
  port: 9300
watcher:
  actions:
    email:
      service:
        account:
          email_account:
            smtp:
              host: prodmail.sing1.com

```

Should be showing those settings? I do see them when I do it on the node level:

curl -XGET [http://kdb006.sing1.com:9200/\_nodes/head001.sing1.com/settings](http://kdb006.sing1.com:9200/_nodes/head001.sing1.com/settings)

```
{
"cluster_name": "prod_sing1",
"nodes": {
"mxBhAVAaQ1S2HlsE9DHzHg": {
"name": "head001.sing1com",
"transport_address": "10.0.0.22:9300",
"host": "10.0.0.2",
"ip": "10.0.0.2",
"version": "2.3.3",
"build": "218bdf1",
"http_address": "10.0.0.2:9200",
"attributes": {
"data": "false",
"master": "true"
},
"settings": {
"index": {
"fielddata": {
"cache": {
"size": "40%"
}
},
"number_of_replicas": "1",
"translog": {
"flush_threshold_ops": "50000"
},
"number_of_shards": "15"
},
"bootstrap": {
"mlockall": "true"
},
"client": {
"type": "node"
},
"gateway": {
"recover_after_nodes": "2"
},
"pidfile": "/var/run/elasticsearch/elasticsearch-prod_sing1.pid",
"network": {
"bind_host": "0.0.0.0",
"publish_host": "10.0.0.2"
},
"threadpool": {
"search": {
"type": "fixed",
"queue_size": "100"
},
"management": {
"size": "20"
},
"index": {
"type": "fixed",
"queue_size": "200"
}
},
"node": {
"data": "false",
"master": "true",
"name": "head001.sing1.com"
},
"http": {
"enabled": "true",
"cors": {
"enabled": "true",
"allow-origin": "/.*/"
}
},
"name": "head001.sing1.com",
"path": {
"data": "/data/warm",
"home": "/usr/share/elasticsearch",
"conf": "/etc/elasticsearch/prod_dwh_expansion",
"logs": "/var/log/elasticsearch/prod_dwh_expansion"
},
"action": {
"disable_delete_all_indices": "1"
},
"config": {
"ignore_system_properties": "true"
},
"cluster": {
"routing": {
"allocation": {
"cluster_concurrent_rebalance": "25",
"disk": {
"watermark": {
"low": "80%",
"high": "90%"
},
"threshold_enabled": "true"
}
}
},
"name": "prod_sing1",
"info": {
"update": {
"interval": "1m"
}
}
},
"indices": {
"memory": {
"index_buffer_size": "50%"
}
},
"watcher": {
"actions": {
"email": {
"service": {
"account": {
"email_account": {
"smtp": {
"host": "prodmail.com"
}
}
}
}
}
}
},
"discovery": {
"zen": {
"minimum_master_nodes": "2",
"ping": {
"unicast": {
"hosts": [
"kdb001.sing1.com",
"kdb002.sing1.com",
"kdb003.sing1.com",
"kdb004.sing1.com",
"kdb005.sing1.com",
"head001.sing1.com",
"head002.sing1.com",
"edge001.sing1.com",
 "edge002.sing1.com"
]
},
"multicast": {
"enabled": "false"
}
}
}
},
"foreground": "false"
}
}
}
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 8, 2016, 12:09am UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/7 "2016-11-08T00:09:33Z")

</div>

> [@fvegaucr](#):
>
> Any suggestion of what I can do we are talking of about 1000+ indices some of them are big as 160GB

How many shards? How many nodes?

> [@fvegaucr](#):
>
> Should be showing those settings?

It does that in 5.0.

---

<div class="post-metadata">

**Author:** ![fvegaucr](https://avatars.discourse-cdn.com/v4/letter/f/7c8e57/32.png) [@fvegaucr](https://discuss.elastic.co/u/fvegaucr)\
**Post date:** [November 8, 2016, 12:25am UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/8 "2016-11-08T00:25:16Z")

</div>

```
"number_of_nodes": 29,
"number_of_data_nodes": 24,
"active_primary_shards": 21076,
"active_shards": 40180,
"relocating_shards": 0,
"initializing_shards": 2,
"active_shards_percent_as_number": 95.32169292085784
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 8, 2016, 12:25am UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/9 "2016-11-08T00:25:57Z")

</div>

> [@fvegaucr](#):
>
> "number\_of\_data\_nodes": 24,  
> "active\_primary\_shards": 21076,  
> "active\_shards": 40180,

The problem is you have too many shards. You need to decrease that, pretty dramatically.

---

<div class="post-metadata">

**Author:** ![fvegaucr](https://avatars.discourse-cdn.com/v4/letter/f/7c8e57/32.png) [@fvegaucr](https://discuss.elastic.co/u/fvegaucr)\
**Post date:** [December 20, 2016, 10:46pm UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/10 "2016-12-20T22:46:37Z")

</div>

Thanks yes we decrease the number

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:04pm UTC](https://discuss.elastic.co/t/unresponsive-cluster/65070/11 "2017-07-05T22:04:44Z")

</div>