Adding a new node to the cluster - shows offline

Elastic Stack Elasticsearch
1

Hey
Added a new node to the cluster.
But it is not getting added?!
Any ideas?
Says it has 5 nodes, but node 5 is offline?

image
image2520×671 57.5 KB

Thanks!

2

GET /_cluster/settings

{
  "persistent" : {
    "indices" : {
      "breaker" : {
        "total" : {
          "limit" : "95%"
        }
      }
    },
    "search" : {
      "max_buckets" : "30000",
      "max_open_scroll_context" : "1000"
    },
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all",
          "exclude" : {
            "_ip" : ""
          }
        }
      }
    }
  }
}
3

http://--IP--:9200


  "name" : "node5",
  "cluster_name" : "cluster_ooo",
  "cluster_uuid" : "O_tH6qaKRHWJ2d1TcWyH1w",
  "version" : {
    "number" : "7.9.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "083627f112ba94dffc1232e8b42b73492789ef91",
    "build_date" : "2020-09-01T21:22:21.964974Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
4

What about the new node and current master logs?

5

@warkolm
In what sense?
Master logs show regular activity.
New logs have no activity in them since I have started it.

6

As in what's in the logs of the node that is showing as offline?

Sometimes Monitoring can take a small amount of time to catch up, but if it's anything longer than a few minutes then your Elasticsearch logs should show something useful.

Also, what's the output from _cat/nodes?v?

7

image
image826×139 9.27 KB

According to logs, the new node is not doing anyhing.
Also, no errors or warnings.

8

Does it have shards on it?

9

I see the data dir is full.

Running stats, shows there is data on the node.
GET _nodes/stats

"indices" : {
        "docs" : {
          "count" : 282316638,
          "deleted" : 1925150
        },
        "store" : {
          "size_in_bytes" : 172751417777,
          "reserved_in_bytes" : 0
        },
        "indexing" : {
          "index_total" : 6559746,
          "index_time_in_millis" : 4369889,
          "index_current" : 2,
          "index_failed" : 0,
          "delete_total" : 1,
          "delete_time_in_millis" : 206,
          "delete_current" : 0,
          "noop_update_total" : 0,
          "is_throttled" : false,
          "throttle_time_in_millis" : 0
        },
        "get" : {
          "total" : 5473258,
          "time_in_millis" : 498471,
          "exists_total" : 4,
          "exists_time_in_millis" : 31,
          "missing_total" : 5473254,
          "missing_time_in_millis" : 498440,
          "current" : 0
        },
        "search" : {
          "open_contexts" : 0,
          "query_total" : 29148,
          "query_time_in_millis" : 697905,
          "query_current" : 0,
          "fetch_total" : 2386,
          "fetch_time_in_millis" : 3554,
          "fetch_current" : 0,
          "scroll_total" : 9232,
          "scroll_time_in_millis" : 1210670,
          "scroll_current" : 0,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0
        },
10

After a few hours, between 2-12 (went to sleep), the status page is reflecting the new node and its metrics.
I expected it to be a quick process, hence I raised this issue.
Thanks!

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.