# Index\_failed number is increasing after adding a new node to elasticsearch cluster(previously single node) **URL:** \ **Category:** Elasticsearch\ **Created:** [March 20, 2023, 4:27pm UTC](https://discuss.elastic.co/t/index-failed-number-is-increasing-after-adding-a-new-node-to-elasticsearch-cluster-previously-single-node/328098 "2023-03-20T16:27:25Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Shreesh\_Narayanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreesh_narayanan/32/87312_2.png) [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Post date:** [March 20, 2023, 4:27pm UTC](https://discuss.elastic.co/t/index-failed-number-is-increasing-after-adding-a-new-node-to-elasticsearch-cluster-previously-single-node/328098/1 "2023-03-20T16:27:25Z")
Hi Folks, Today i added a new node to a previously single -node elasticsearch cluster and the process was successful . however when i look at the node stats (via the node stats API) it shows the index\_failed numbers to be increasing , i don't see anything alarming in the logs though . Currently i just have 2 nodes in the elastic cluster, a third node is being provisioned and will be ready in probably the next week. I will be adding that node to this cluster once the node is all set. ```auto GET /_nodes/K0giPxyJRO-sV9-x-mOH9w/stats { "_nodes": { "total": 1, "successful": 1, "failed": 0 }, "cluster_name": "Nexus-Prod", "nodes": { "K0giPxyJRO-sV9-x-mOH9w": { "timestamp": 1679328488501, "name": "nexus-elasticsearch-node2", "transport_address": "10.27.101.247:9300", "host": "10.27.101.247", "ip": "10.27.101.247:9300", "roles": [ "data", "master" ], "attributes": { "xpack.installed": "true" }, "indices": { "docs": { "count": 40846394, "deleted": 3391312 }, "shard_stats": { "total_count": 105 }, "store": { "size_in_bytes": 15360584112, "total_data_set_size_in_bytes": 15360584112, "reserved_in_bytes": 0 }, "indexing": { "index_total": 135497, "index_time_in_millis": 23617, "index_current": 0, "index_failed": 87703, "delete_total": 520, "delete_time_in_millis": 9, "delete_current": 0, "noop_update_total": 0, "is_throttled": false, "throttle_time_in_millis": 0, "write_load": 0.00005004396408600462 }, "get": { "total": 9003, "time_in_millis": 1576, "exists_total": 8745, "exists_time_in_millis": 1556, "missing_total": 258, "missing_time_in_millis": 20, "current": 0 }, [AFTER A FEW SECONDS] }, "indexing": { "index_total": 142209, "index_time_in_millis": 24872, "index_current": 0, "index_failed": 92389, "delete_total": 520, "delete_time_in_millis": 9, "delete_current": 0, "noop_update_total": 0, "is_throttled": false, "throttle_time_in_millis": 0, "write_load": 0.00005022018880384734 }, ``` Here;s the log from the new node , i dont see any errors ```auto [2023-03-20T14:59:16,172][INFO][o.e.c.s.ClusterApplierService] [nexus-elasticsearch-node2] master node changed {previous [], current [{nexus-elasticsearch-node1}{Xf9i6l_JQbiBXvhzw_fxzg}{tL951vCeQdGpNgdir_qyZw}{nexus-elasticsearch-node1}{10.27.101.63}{10.27.101.63:9300}{dm}]}, removed {{nexus-elasticsearch-node1}{Xf9i6l_JQbiBXvhzw_fxzg}{9i0P-GkwRLuTl6wRz_1JJg}{nexus-elasticsearch-node1}{10.27.101.63}{10.27.101.63:9300}{dm}}, added {{nexus-elasticsearch-node1}{Xf9i6l_JQbiBXvhzw_fxzg}{tL951vCeQdGpNgdir_qyZw}{nexus-elasticsearch-node1}{10.27.101.63}{10.27.101.63:9300}{dm}}, term: 76, version: 19412, reason: ApplyCommitRequest{term=76, version=19412, sourceNode={nexus-elasticsearch-node1}{Xf9i6l_JQbiBXvhzw_fxzg}{tL951vCeQdGpNgdir_qyZw}{nexus-elasticsearch-node1}{10.27.101.63}{10.27.101.63:9300}{dm}{xpack.installed=true}} [2023-03-20T14:59:16,659][INFO][o.e.x.s.a.TokenService] [nexus-elasticsearch-node2] refresh keys [2023-03-20T14:59:16,810][INFO][o.e.x.s.a.TokenService] [nexus-elasticsearch-node2] refreshed keys ``` Here's the allocation stats: ```auto shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 105 14.3gb 19.4gb 73.5gb 92.9gb 20 10.27.101.247 10.27.101.247 nexus-elasticsearch-node2 104 13.7gb 19.1gb 78.6gb 97.8gb 19 10.27.101.63 10.27.101.63 nexus-elasticsearch-node1 ``` No unassigned shards either ```auto root@prod-nexus-app06:~# curl -k -XGET -u elastic:{$REDACTED} 'https://10.27.101.247:9200/_cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state' 2>&1 | grep UNASSIGN root@prod-nexus-app06:~# ``` ---
**Author:** ![William\_Brafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/william_brafford/32/51559_2.png) [@William\_Brafford](https://discuss.elastic.co/u/William_Brafford)\ **Post date:** [March 20, 2023, 6:25pm UTC](https://discuss.elastic.co/t/index-failed-number-is-increasing-after-adding-a-new-node-to-elasticsearch-cluster-previously-single-node/328098/2 "2023-03-20T18:25:42Z")
Hi, @Shreesh_Narayanan . The `index_failed` stat you are looking at has to do with "indexing operations," meaning whether a document is successfully indexed into Elasticsearch or not. A perfectly healthy index might not be able to index certain documents for many reasons: perhaps a field in the document doesn't match the index's mappings, or perhaps dynamic mappings are disabled and a document contains an unexpected field. If you want to find out what these errors are, I would look at whatever tool you are using for ingest. A response from the Elasticsearch bulk API provides information on indexing failures, so hopefully whatever is sending documents to Elasticsearch is logging information about indexing failures. There may be a misconfiguration or a mapping issue to troubleshoot, but the node itself seems to be healthy. ---
**Author:** ![Shreesh\_Narayanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreesh_narayanan/32/87312_2.png) [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Post date:** [March 21, 2023, 5:32am UTC](https://discuss.elastic.co/t/index-failed-number-is-increasing-after-adding-a-new-node-to-elasticsearch-cluster-previously-single-node/328098/3 "2023-03-21T05:32:45Z")
thank you for the response, as of now i don't see any errors and data pops up in kibana with no problem ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 18, 2023, 5:32am UTC](https://discuss.elastic.co/t/index-failed-number-is-increasing-after-adding-a-new-node-to-elasticsearch-cluster-previously-single-node/328098/4 "2023-04-18T05:32:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.