# New documents are not indexed/searchable in the cluster when 2 nodes are off out of 3 **URL:** \ **Category:** Elasticsearch\ **Created:** [April 25, 2020, 10:12am UTC](https://discuss.elastic.co/t/new-documents-are-not-indexed-searchable-in-the-cluster-when-2-nodes-are-off-out-of-3/229750 "2020-04-25T10:12:49Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Vano\_Maisuradze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vano_maisuradze/32/63460_2.png) [@Vano\_Maisuradze](https://discuss.elastic.co/u/Vano_Maisuradze)\ **Post date:** [April 25, 2020, 10:12am UTC](https://discuss.elastic.co/t/new-documents-are-not-indexed-searchable-in-the-cluster-when-2-nodes-are-off-out-of-3/229750/1 "2020-04-25T10:12:49Z")
I have 3 nodes in the cluster. All of them are master and data nodes. When all nodes are online, indexing works fine. If I turn off only one node, it is still operating normally, but when 2 nodes goes offline, indexing works (getting successful result), but indexed documents are not searchable (Also number of documents are not updating. E.g. by running _/test\_index/\_count_ returns the old value). My goal is to have a cluster with 3 nodes, where if two of them goes offline, I must be able to index and query from 3rd node. Any ideas why this could happen and how to achieve this goal? **Here is the technical details:** _Number of shards:_ 15 _Number of replicas:_ 2 **node-1 config:** ``` cluster.name: "cluster_name" node.name: "node-1" node.master: true node.data: true network.host: [_local_, "10.0.2.170"] discovery.seed_hosts: ["10.0.2.170", "10.0.2.171", "10.0.2.172"] action.auto_create_index: "*" cluster.initial_master_nodes: ["node-1", "node-2", "node-3"] ``` **node-2 config:** ``` cluster.name: "cluster_name" node.name: "node-2" node.master: true node.data: true network.host: [_local_, "10.0.2.171"] discovery.seed_hosts: ["10.0.2.170", "10.0.2.171", "10.0.2.172"] action.auto_create_index: "*" cluster.initial_master_nodes: ["node-1", "node-2", "node-3"] ``` **node-3 config:** ``` cluster.name: "cluster_name" node.name: "node-3" node.master: true node.data: true network.host: [_local_, "10.0.2.172"] discovery.seed_hosts: ["10.0.2.170", "10.0.2.171", "10.0.2.172"] action.auto_create_index: "*" cluster.initial_master_nodes: ["node-1", "node-2", "node-3"] ``` _All of them are running on AWS t2.micro servers_ When 2 nodes are offline and try to get cluster health info from the 3rd node (using \_cluster/health), I'm getting: ``` { "error" : { "root_cause" : [ { "type" : "master_not_discovered_exception", "reason" : null } ], "type" : "master_not_discovered_exception", "reason" : null }, "status" : 503 } ``` Using \_cluster/stats, I'm getting: ``` { "_nodes" : { "total" : 3, "successful" : 1, "failed" : 2, "failures" : [ { "type" : "failed_node_exception", "reason" : "Failed node [cQ8Z2v3TSFeF8eXs-OfIyw]", "node_id" : "cQ8Z2v3TSFeF8eXs-OfIyw", "caused_by" : { "type" : "node_not_connected_exception", "reason" : "[node-2][10.0.2.171:9300] Node not connected" } }, { "type" : "failed_node_exception", "reason" : "Failed node [boNPPrk-SaWSOtAx_ZfwMA]", "node_id" : "boNPPrk-SaWSOtAx_ZfwMA", "caused_by" : { "type" : "node_not_connected_exception", "reason" : "[node-3][10.0.2.172:9300] Node not connected" } } ] } ``` If one of the offline nodes are turned on, same queries return following results (Now we have 2 available nodes): \_cluster/health: ``` { "cluster_name" : "cluster_name", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 20, "active_shards" : 40, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 15, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 72.72727272727273 } ``` \_cluster/stats: ``` { "_nodes" : { "total" : 2, "successful" : 2, "failed" : 0 } ``` If 3rd node becomes available (Now we have all 3 nodes online): \_cluster/health ``` { "cluster_name" : "cluster_name", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 20, "active_shards" : 55, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` \_cluster/stats ``` { "_nodes" : { "total" : 3, "successful" : 3, "failed" : 0 } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 25, 2020, 10:36am UTC](https://discuss.elastic.co/t/new-documents-are-not-indexed-searchable-in-the-cluster-when-2-nodes-are-off-out-of-3/229750/2 "2020-04-25T10:36:20Z")
A majority of master eligible nodes always need to be available in order to have a fully functioning cluster, so with three nodes you can only tolerate one node being offline. What you describe is therefore not possible. ---
**Author:** ![Vano\_Maisuradze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vano_maisuradze/32/63460_2.png) [@Vano\_Maisuradze](https://discuss.elastic.co/u/Vano_Maisuradze)\ **Post date:** [April 25, 2020, 10:42am UTC](https://discuss.elastic.co/t/new-documents-are-not-indexed-searchable-in-the-cluster-when-2-nodes-are-off-out-of-3/229750/3 "2020-04-25T10:42:37Z")
Thanks for the quick reply. So it means, I can't afford to lose two nodes, but this guide says I can: [Replica Shards | Elasticsearch: The Definitive Guide [2.x] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/current/replica-shards.html#_balancing_load_with_replicas) > As a bonus, we have also increased our availability. We can now afford to lose two nodes and still have a copy of all our data. We are not losing old data, but if I can't index new documents while 2 nodes are offline, it also means that I lost new ones. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 25, 2020, 10:44am UTC](https://discuss.elastic.co/t/new-documents-are-not-indexed-searchable-in-the-cluster-when-2-nodes-are-off-out-of-3/229750/4 "2020-04-25T10:44:15Z")
You can lose two nodes without losing data, but you will not have a fully functioning cluster that can index or update data. Indexing data requires an active master node in order to prevent data loss and that requires a majority to be available. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 23, 2020, 10:44am UTC](https://discuss.elastic.co/t/new-documents-are-not-indexed-searchable-in-the-cluster-when-2-nodes-are-off-out-of-3/229750/5 "2020-05-23T10:44:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.