# Dec 21st, 2021: \[en\] Top 3 recommendations to keep your Elasticsearch cluster healthy **URL:** \ **Category:** Advent Calendar\ **Created:** [December 21, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-21st-2021-en-top-3-recommendations-to-keep-your-elasticsearch-cluster-healthy/289326 "2021-12-21T08:00:53Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Imma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imma/32/32013_2.png) [@Imma](https://discuss.elastic.co/u/Imma)\ **Post date:** [December 21, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-21st-2021-en-top-3-recommendations-to-keep-your-elasticsearch-cluster-healthy/289326/1 "2021-12-21T08:00:53Z")
This article is also available in Spanish. > [@Dec 21st, 2021: \[es\] Tres recomendaciones para tener un clúster Elasticsearch saludable](https://discuss.elastic.co/t/289328): > > This article is also available in English: [https://discuss.elastic.co/t/289326](https://discuss.elastic.co/t/289326) En nuestro día a día de soporte en Elastic vemos algunas herramientas que son muy útiles cuando necesitamos ayudar a nuestros clientes a solucionar problemas en sus clústeres de Elasticsearch o monitorizar para mantenerlos saludables. Repasemos las 3 principales con algunos ejemplos. 1. Conoce tus APIs REST Conocer las [APIs REST](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/rest-apis.html) de Elasticsearch es muy útil para manter el clúster en buen estado. No solo ayudan a so… Working as an Elastic support engineer, we see a few tools that are very useful to have already in place when we need to help our customers troubleshoot their Elasticsearch clusters, or monitor to keep them healthy. Let’s review the top 3 with a few examples. #### 1. Know your REST APIs Knowing Elasticsearch [REST APIs](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/rest-apis.html) is very useful to keep your cluster healthy. Not only can they help you troubleshoot, but prevent issues. If you want something more human-readable, have a look at the [CAT APIs](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/cat.html). The first thing to keep the cluster healthy is to keep it in green health. With a simple call to the [cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/cluster-health.html): ```auto GET /_cluster/health ``` We’ll get an overview of our cluster status. ```json { "cluster_name": "eyeveebee-prod-cluster", "status": "red", "timed_out": false, "number_of_nodes": 30, "number_of_data_nodes": 27, "active_primary_shards": 15537, "active_shards": 26087, "relocating_shards": 0, "initializing_shards": 0, "unassigned_shards": 1, "delayed_unassigned_shards": 0, "number_of_pending_tasks": 209, "number_of_in_flight_fetch": 0, "task_max_waiting_in_queue_millis": 235940, "active_shards_percent_as_number": 99.99616762873977 } ``` In the case above, we have a cluster in red status. We can see we have one unassigned shard, which is causing the cluster to be red. The health of a cluster is that of the worst shard of the worst index. Therefore, at least some index or shard will be in red. It’s important that we keep our cluster in green health. [Kibana alerts](https://www.elastic.co/guide/en/kibana/7.16/kibana-alerts.html#kibana-alerts-cluster-alerts) can help us here, notifying us when the cluster becomes yellow (missing at least one replica shard) or red (missing at least one primary shard). To further investigate what index is red, we can use the [CAT indices API](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/cat-indices.html): ```auto GET _cat/indices?v&s=health:desc,index&h=health,status,index,docs.count,pri,rep ``` Where we could locate the red index. ```auto health status index docs.count pri rep red open eventlogs-000007 1 1 green open .apm-agent-configuration 0 1 1 ... ``` With the [CAT](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-shards.html) [shards](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/cat-shards.html) [API](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-shards.html) we can have look at the shards for the red index ‘eventlogs-000007’: ```auto GET /_cat/shards?v&s=state:asc,node,index&h=index,shard,prirep,state,docs,node ``` Where we would be able to determine that indeed we are missing a primary shard, which is ‘UNASSIGNED’. ```auto index shard prirep state docs node eventlogs-000007 0 p UNASSIGNED .apm-agent-configuration 0 p STARTED 0 instance-0000000012 ... ``` Finally, we can use the [Cluster Allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/cluster-allocation-explain.html) to find the reason why. ```auto GET _cluster/allocation/explain { "index": "eventlogs-000007", "shard": 0, "primary": true } ``` We would get an explanation similar to the following, which should allow us to get to the root cause. ```json { "index" : "eventlogs-000007", "shard" : 0, "primary" : false, "current_state" : "unassigned", "unassigned_info" : { "reason" : "NODE_LEFT", "at" : "2021-12-08T17:00:53.596Z", "details" : "node_left [gyv9cseHQyWD-FjLTfSnvA]", "last_allocation_status" : "no_attempt" }, "can_allocate" : "no", "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions" : [ { "node_id" : "-PYVqLCLTSKjriA6UZthuw", "node_name" : "instance-0000000012", "transport_address" : "10.43.1.6:19294", "node_attributes" : { "xpack.installed" : "true", "data" : "hot", "transform.node" : "true" }, "node_decision" : "no", "deciders" : [ { "decider" : "same_shard", "decision" : "NO", "explanation" : "the node is above the high watermark cluster setting [cluster.routing.allocation.disk.watermark.high=90.0%], having less than the minimum required [90.0%] free space, actual free: [9.13%]" } ] }, ... ``` In this case, we ran out of storage on our nodes and shards cannot be assigned. Something we could further confirm using the [CAT allocation API](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/cat-allocation.html). We can prevent hitting the [disk watermarks](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/modules-cluster.html#disk-based-shard-allocation) with [Kibana alerts for disk usage threshold](https://www.elastic.co/guide/en/kibana/7.16/kibana-alerts.html#kibana-alerts-disk-usage-threshold). If we keep our nodes around 75% storage, we will have room for growth and keep our cluster healthier. Starting at 85% storage used (default value), we’ll have shard allocation limitations. We can’t stress enough the importance of planning for data retention. [Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/index-lifecycle-management.html) and the use of [Data Tiers](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/data-tiers.html) are of great help to keep storage in check. Also good to read the documentation on [Size your shards](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/size-your-shards.html#size-your-shards), which is also key in keeping a healthy cluster. Don’t forget to use Elasticsearch [snapshots](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/snapshot-restore-apis.html), with [snapshot lifecycle-management](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/snapshot-lifecycle-management.html), to prepare for scenarios where we could lose data if we have no backup. And to [set up for high availability](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/high-availability.html). We also observe in the cluster health above that the number of pending tasks could be a bit high. And we could further look at those again with a call to the [CAT pending tasks API](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/cat-pending-tasks.html): ```auto GET /_cat/pending_tasks?v ``` To find out what tasks we have pending. ```auto insertOrder timeInQueue priority source 412717 5.2s NORMAL restore_snapshot[2021.12.17-.ds-logs-2021.12.09-000030-trdkfuewwkjaca] 412718 2s NORMAL ilm-execute-cluster-state-steps [{"phase":"cold","action":"searchable_snapshot","name":"wait-for-index-color"} => {"phase":"cold","action":"searchable_snapshot","name":"copy-execution-state"}] ... ``` Or we could also use [jq](https://stedolan.github.io/jq/) to aggregate the results of the [pending cluster tasks API](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/cluster-pending.html) to more easily investigate what are the tasks we have pending. ```auto curl --silent --compressed 'https://localhost:9200/_cluster/pending_tasks' | jq '.tasks[].source' -cMr | sed -e 's/[.*//' | sort | uniq -c ``` Which could give us a better idea of what is causing pending tasks: ```auto 1 restore_snapshot 17 delete-index 183 ilm-execute-cluster-state-steps 2 node-join 4 update task state 2 update-settings ``` This is just to showcase that knowing the REST APIs available we can get very helpful information to assess our cluster’s health and adjust our architecture accordingly. Finally, have a peek at [Elastic’s support diagnostics](https://github.com/elastic/support-diagnostics). Those are the REST APIs calls that we use at Elastic Support to help our customers keep their clusters healthy. Or have a look at our blog “[Why does Elastic support keep asking for diagnostic files](https://www.elastic.co/blog/why-does-elastic-support-keep-asking-for-diagnostic-files)” which explains the underlying reasons why. #### 2. Take Advantage of the Stack Monitoring & Alerting The second recommendation is to plan for a [separate monitoring cluster](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/monitoring-production.html) when in production. The REST APIs give us current information, but we are missing the historic data. If we send that to a monitoring cluster, it will help us investigate incidents, forecast capacity, etc. [Kibana alerts for the Elastic Stack monitoring](https://www.elastic.co/guide/en/kibana/7.16/kibana-alerts.html) will notify us of potential issues. One example we see a lot in Elastic support, where monitoring comes in handy, is node hot-spotting. If we have a cluster that is showing high CPU, let’s say, during ingestion, on one or just a few of the data nodes, while the others are idle; and those nodes keep changing. We can use monitoring to confirm our suspicions. Let’s have a look at the [Kibana Stack Monitoring](https://www.elastic.co/guide/en/kibana/7.16/xpack-monitoring.html) UI for our cluster. Out of 3 nodes, 1 is showing high CPU usage. ![](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5fac5df5de207740be2d47b1358ab146ae012b6c.png) We could investigate further by going to the indices tab. We might find, like in this case, that during the window when we see high CPU usage on one node, we have an index ‘log-201998’ that had a very high ingest rate compared to the rest. ![](https://us1.discourse-cdn.com/elastic/original/3X/f/7/f772f426d8f283ec4f8d81d63a398301a9fcb9fc.png) If this index has one primary shard, and it’s the only one with a high ingest rate, we could assign 3 primary shards, so the load is balanced between the 3 data instances we have in this example. For bigger clusters and more than one hot index, the situation might be not so straightforward. We might need to limit the number of shards for those indices that end up on each cluster node. Check on our docs to [avoid node hotspots](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/size-your-shards.html#avoid-node-hotspots). Having a monitoring cluster will be of great help. #### 3. Proactively check Logs One last recommendation is to proactively review the logs. We can use [Filebeat’s Elasticsearch module](https://www.elastic.co/guide/en/beats/filebeat/7.16/filebeat-module-elasticsearch.html) to ingest our logs in the monitoring cluster we discussed in the previous section. And even use the stack capabilities to [categorize logs](https://www.elastic.co/guide/en/observability/7.16/categorize-logs.html) to discover anything abnormal and alert us. One example we see with our customers a lot is wrong mapping [data types](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/mapping-types.html) in the indices. Depending on how we configure our index mappings, we might be losing documents that come with conflicting types. If we check our cluster logs, we would see those errors and be able to act. Let’s take the example of a document that sometimes has a numeric value in the source, and sometimes it’s alphanumeric. If we use the default [dynamic mappings](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/mapping.html#mapping-dynamic), and we first ingest this document with a numeric value of a field we’ll call “key”: ```auto POST my-test-index/_doc { "key": 0 } ``` Elasticsearch will interpret this field as a [number](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/number.html), of type long. ```auto GET my-test-index/_mapping/field/key ``` ```json { "my-test-index" : { "mappings" : { "key" : { "full_name" : "key", "mapping" : { "key" : { "type" : "long" } } } } } } ``` If the next document came with, let’s say, with a UUID: ```auto POST my-test-index/_doc { "key": "123e4567-e89b-12d3-a456-426614174000" } ``` In Kibana we would see the error. ```json { "error" : { "root_cause" : [ { "type" : "mapper_parsing_exception", "reason" : "failed to parse field [key] of type [long] in document with id '4Wtyz30BtaU7QP7QuSQY'. Preview of field's value: '123e4567-e89b-12d3-a456-426614174000'" } ], "type" : "mapper_parsing_exception", "reason" : "failed to parse field [key] of type [long] in document with id '4Wtyz30BtaU7QP7QuSQY'. Preview of field's value: '123e4567-e89b-12d3-a456-426614174000'", "caused_by" : { "type" : "illegal_argument_exception", "reason" : "For input string: "123e4567-e89b-12d3-a456-426614174000"" } }, "status" : 400 } ``` This status code 400 is a non-retriable error, and it means Elasticsearch will not index the document, and clients like Logstash or Agent won’t retry. If we search for our documents on the index, we only have the first one. ```auto GET my-test-index/_search?filter_path=hits.total,hits.hits._source ``` ```json { "hits" : { "total" : { "value" : 1, "relation" : "eq" }, "hits" : [ { "_source" : { "key" : 0 } } ] } } ``` This that we see in Kibana, would appear in our logs too. It’s very common to find this in Logstash (logs we can also ingest using [Filebeat’s Logstash module](https://www.elastic.co/guide/en/beats/filebeat/7.16/filebeat-module-logstash.html)). It could go unnoticed unless we check our logs. ```auto Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"my-test-index-0000001", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"my-test-index-0000001", "_type"=>"_doc", "_id"=>"Qmt6z30BtaU7QP7Q4SXE", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [key] of type [long] in document with id 'Qmt6z30BtaU7QP7Q4SXE' ``` As a bonus, if we ingest logs, we will be ready to discover [slow logs](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/index-modules-slowlog.html) in case we need to troubleshoot any search or ingest slowness. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2022, 8:01am UTC](https://discuss.elastic.co/t/dec-21st-2021-en-top-3-recommendations-to-keep-your-elasticsearch-cluster-healthy/289326/2 "2022-01-18T08:01:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.