# Cant get cluster health

**URL:** <https://discuss.elastic.co/t/cant-get-cluster-health/255405>\
**Category:** Kibana\
**Tags:** elastic-stack-monitoring\
**Created:** [November 14, 2020, 12:31pm UTC](https://discuss.elastic.co/t/cant-get-cluster-health/255405 "2020-11-14T12:31:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\
**Post date:** [November 14, 2020, 12:31pm UTC](https://discuss.elastic.co/t/cant-get-cluster-health/255405/1 "2020-11-14T12:31:20Z")

</div>

Hello everyone!

A couple of days ago, the monitoring cluster suddenly broke down. It just stopped showing the latest data at some point. Message standard

 ![photo_2020-11-14_15-00-13](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0e64d21ada825747966fa1c129ade195fe7186ca.jpeg)

When you try to enable the built-in monitor nothing happens

 ![photo_2020-11-14_15-00-14](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cb0e65feb7271744c0fbaa54168bcefe3f8365d5.jpeg)

If you select interval in the last 7 days and try Set up monitoring with Metricbeat, you can see the old data and the error cluster

 ![photo_2020-11-14_15-00-14 (2)](https://us1.discourse-cdn.com/elastic/original/3X/0/9/09004c7ab44bd92f24da308fd7b45dc540a7f31c.jpeg)

If the pre-change interval for the last hour, error of the cluster doesnt show up, but metrics like still not come

 ![photo_2020-11-14_15-00-14 (3)](https://us1.discourse-cdn.com/elastic/original/3X/1/3/13b49ccdf083ae436a9e3b267c4a90529f886927.jpeg)

I set up Metricbeat, everything seems to be working fine. In any case, everything starts and seems to transmit data

 ![photo_2020-11-14_15-00-14 (4)](https://us1.discourse-cdn.com/elastic/original/3X/6/1/618eccac6fd87c676123e6a84f80eb42c3dc7cc7.jpeg)

Metrics indicies are exist

 ![Снимок](https://us1.discourse-cdn.com/elastic/original/3X/4/8/480c011cb25b70cec9b57014b2fe20e16faf7654.png)

ES cluster and KIbana versions are 7.9.1, Metricbeat 7.10

Please tell me what it can be and in which direction to look?

---

<div class="post-metadata">

**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\
**Post date:** [November 14, 2020, 1:24pm UTC](https://discuss.elastic.co/t/cant-get-cluster-health/255405/2 "2020-11-14T13:24:50Z")

</div>

I tried

```
PUT _settings
 {
  "index": {
    "blocks": {
      "read_only_allow_delete": "false"
     }
   }
 }

```

Nothing changed. By the way, logs are coming in indicies without any problems, so there is no free space problems.

Metricbeat.yml:

```
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana:
  # Kibana Host
  host: "https://10.199.5.107:5601/"

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://10.199.5.104:9200", "https://10.199.5.105:9200", "https://10.199.5.106:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "qTwQGayENAo1fbVpepzt"
  ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca.crt"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

monitoring.enabled: true

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

```

elasticsearch-xpack.yml:

```
- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - enrich
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://10.199.5.104:9200"]
  username: "elastic"
  password: "qTwQGayENAo1fbVpepzt"
  ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca.crt"]
  xpack.enabled: true
```

---

<div class="post-metadata">

**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\
**Post date:** [November 14, 2020, 7:11pm UTC](https://discuss.elastic.co/t/cant-get-cluster-health/255405/3 "2020-11-14T19:11:38Z")

</div>

Sorry, forgot about one thing. A couple of months ago i've updated cluster from 7.1 to 7.9.1. Three days ago i noticed that old index lifecycle policices doesnt work anymore and created new ILM policy named [delete\_old\_indicies].

I tried  
cat /var/log/elasticsearch/elk-cluster.mosgortrans.com.log | grep errors and there is nothing. cat /var/log/elasticsearch/elk-cluster.mosgortrans.com.log | grep monit show this

```
[2020-11-14T20:14:22,336][WARN][o.e.x.i.IndexLifecycleRunner] [elk-es-node-03] current step [null] for index [.monitoring-kibana-7-2020.11.13] with policy [delete_old_indicies] is not recognized
[2020-11-14T20:14:22,361][WARN][o.e.x.i.IndexLifecycleRunner] [elk-es-node-03] current step [null] for index [.monitoring-logstash-7-2020.11.13] with policy [delete_old_indicies] is not recognized
[2020-11-14T20:14:22,392][WARN][o.e.x.i.IndexLifecycleRunner] [elk-es-node-03] current step [null] for index [.monitoring-es-7-2020.11.14] with policy [delete_old_indicies] is not recognized
[2020-11-14T20:14:22,424][WARN][o.e.x.i.IndexLifecycleRunner] [elk-es-node-03] current step [null] for index [.monitoring-es-7-2020.11.12] with policy [delete_old_indicies] is not recognized
[2020-11-14T20:14:22,447][WARN][o.e.x.i.IndexLifecycleRunner] [elk-es-node-03] current step [null] for index [.monitoring-kibana-7-2020.11.14] with policy [delete_old_indicies] is not recognized
[2020-11-14T20:14:22,461][WARN][o.e.x.i.IndexLifecycleRunner] [elk-es-node-03] current step [null] for index [.monitoring-es-7-2020.11.13] with policy [delete_old_indicies] is not recognized
[2020-11-14T20:14:22,514][WARN][o.e.x.i.IndexLifecycleRunner] [elk-es-node-03] current step [null] for index [.monitoring-beats-7-2020.11.14] with policy [delete_old_indicies] is not recognized
[2020-11-14T20:14:22,531][WARN][o.e.x.i.IndexLifecycleRunner] [elk-es-node-03] current step [null] for index [.monitoring-es-7-mb-2020.11.14] with policy [delete_old_indicies] is not recognized
[2020-11-14T20:14:25,193][INFO][o.e.i.s.IndexShard] [elk-es-node-03] [.monitoring-logstash-7-2020.11.13][1] primary-replica resync completed with 0 operations
[2020-11-14T20:14:26,291][INFO][o.e.i.s.IndexShard] [elk-es-node-03] [.monitoring-es-7-2020.11.11][0] primary-replica resync completed with 0 operations
[2020-11-14T20:14:26,299][INFO][o.e.i.s.IndexShard] [elk-es-node-03] [.monitoring-es-7-2020.11.14][0] primary-replica resync completed with 0 operations
[2020-11-14T20:14:26,979][WARN][o.e.a.b.TransportShardBulkAction] [elk-es-node-03] [[.monitoring-es-7-mb-2020.11.14][1]] failed to perform indices:data/write/bulk[s] on replica [.monitoring-es-7-mb-2020.11.14][1], node[IvxKqzsjRmCnOwJNqsLOQg], [R], s[STARTED], a[id=UnXFmesbR56lC-K-GBx66A]
[2020-11-14T20:14:27,386][WARN][o.e.c.r.a.AllocationService] [elk-es-node-03] [.monitoring-es-7-mb-2020.11.14][0] marking unavailable shards as stale: [zE8Q1fIbQ2ag55TqVmgCTg]
[2020-11-14T20:14:27,386][WARN][o.e.c.r.a.AllocationService] [elk-es-node-03] [.monitoring-es-7-mb-2020.11.14][1] marking unavailable shards as stale: [UnXFmesbR56lC-K-GBx66A]
[2020-11-14T20:14:27,730][WARN][o.e.c.r.a.AllocationService] [elk-es-node-03] [.monitoring-kibana-7-2020.11.14][1] marking unavailable shards as stale: [Eb2VQq_jSYayqdSwtaNnJw]
[2020-11-14T20:14:29,801][WARN][o.e.c.r.a.AllocationService] [elk-es-node-03] [.monitoring-beats-7-2020.11.14][1] marking unavailable shards as stale: [Dj8qZSUER2K9MT5hO6yibQ]
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 12, 2020, 7:11pm UTC](https://discuss.elastic.co/t/cant-get-cluster-health/255405/4 "2020-12-12T19:11:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.