Index_failed number is increasing after adding a new node to elasticsearch cluster(previously single node)

Elastic Stack Elasticsearch
1

Hi Folks,

Today i added a new node to a previously single -node elasticsearch cluster and the process was successful . however when i look at the node stats (via the node stats API) it shows the index_failed numbers to be increasing , i don't see anything alarming in the logs though .

Currently i just have 2 nodes in the elastic cluster, a third node is being provisioned and will be ready in probably the next week. I will be adding that node to this cluster once the node is all set.

GET /_nodes/K0giPxyJRO-sV9-x-mOH9w/stats

{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "Nexus-Prod",
  "nodes": {
    "K0giPxyJRO-sV9-x-mOH9w": {
      "timestamp": 1679328488501,
      "name": "nexus-elasticsearch-node2",
      "transport_address": "10.27.101.247:9300",
      "host": "10.27.101.247",
      "ip": "10.27.101.247:9300",
      "roles": [
        "data",
        "master"
      ],
      "attributes": {
        "xpack.installed": "true"
      },
      "indices": {
        "docs": {
          "count": 40846394,
          "deleted": 3391312
        },
        "shard_stats": {
          "total_count": 105
        },
        "store": {
          "size_in_bytes": 15360584112,
          "total_data_set_size_in_bytes": 15360584112,
          "reserved_in_bytes": 0
        },
        "indexing": {
          "index_total": 135497,
          "index_time_in_millis": 23617,
          "index_current": 0,
          "index_failed": 87703,
          "delete_total": 520,
          "delete_time_in_millis": 9,
          "delete_current": 0,
          "noop_update_total": 0,
          "is_throttled": false,
          "throttle_time_in_millis": 0,
          "write_load": 0.00005004396408600462
        },
        "get": {
          "total": 9003,
          "time_in_millis": 1576,
          "exists_total": 8745,
          "exists_time_in_millis": 1556,
          "missing_total": 258,
          "missing_time_in_millis": 20,
          "current": 0
        },

[AFTER A FEW SECONDS]

 },
        "indexing": {
          "index_total": 142209,
          "index_time_in_millis": 24872,
          "index_current": 0,
          "index_failed": 92389,
          "delete_total": 520,
          "delete_time_in_millis": 9,
          "delete_current": 0,
          "noop_update_total": 0,
          "is_throttled": false,
          "throttle_time_in_millis": 0,
          "write_load": 0.00005022018880384734
        },

Here;s the log from the new node , i dont see any errors


[2023-03-20T14:59:16,172][INFO ][o.e.c.s.ClusterApplierService] [nexus-elasticsearch-node2] master node changed {previous [], current [{nexus-elasticsearch-node1}{Xf9i6l_JQbiBXvhzw_fxzg}{tL951vCeQdGpNgdir_qyZw}{nexus-elasticsearch-node1}{10.27.101.63}{10.27.101.63:9300}{dm}]}, removed {{nexus-elasticsearch-node1}{Xf9i6l_JQbiBXvhzw_fxzg}{9i0P-GkwRLuTl6wRz_1JJg}{nexus-elasticsearch-node1}{10.27.101.63}{10.27.101.63:9300}{dm}}, added {{nexus-elasticsearch-node1}{Xf9i6l_JQbiBXvhzw_fxzg}{tL951vCeQdGpNgdir_qyZw}{nexus-elasticsearch-node1}{10.27.101.63}{10.27.101.63:9300}{dm}}, term: 76, version: 19412, reason: ApplyCommitRequest{term=76, version=19412, sourceNode={nexus-elasticsearch-node1}{Xf9i6l_JQbiBXvhzw_fxzg}{tL951vCeQdGpNgdir_qyZw}{nexus-elasticsearch-node1}{10.27.101.63}{10.27.101.63:9300}{dm}{xpack.installed=true}}
[2023-03-20T14:59:16,659][INFO ][o.e.x.s.a.TokenService   ] [nexus-elasticsearch-node2] refresh keys
[2023-03-20T14:59:16,810][INFO ][o.e.x.s.a.TokenService   ] [nexus-elasticsearch-node2] refreshed keys

Here's the allocation stats:

shards disk.indices disk.used disk.avail disk.total disk.percent host          ip            node
   105       14.3gb    19.4gb     73.5gb     92.9gb           20 10.27.101.247 10.27.101.247 nexus-elasticsearch-node2
   104       13.7gb    19.1gb     78.6gb     97.8gb           19 10.27.101.63  10.27.101.63  nexus-elasticsearch-node1

No unassigned shards either


root@prod-nexus-app06:~# curl -k -XGET -u elastic:{$REDACTED} 'https://10.27.101.247:9200/_cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state'  2>&1 |  grep UNASSIGN
root@prod-nexus-app06:~#
2

Hi, @Shreesh_Narayanan . The index_failed stat you are looking at has to do with "indexing operations," meaning whether a document is successfully indexed into Elasticsearch or not. A perfectly healthy index might not be able to index certain documents for many reasons: perhaps a field in the document doesn't match the index's mappings, or perhaps dynamic mappings are disabled and a document contains an unexpected field.

If you want to find out what these errors are, I would look at whatever tool you are using for ingest. A response from the Elasticsearch bulk API provides information on indexing failures, so hopefully whatever is sending documents to Elasticsearch is logging information about indexing failures. There may be a misconfiguration or a mapping issue to troubleshoot, but the node itself seems to be healthy.

1 Like
3

thank you for the response, as of now i don't see any errors and data pops up in kibana with no problem

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.