# Heterogeneous Cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [September 7, 2024, 7:53am UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184 "2024-09-07T07:53:52Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\ **Post date:** [September 7, 2024, 7:53am UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/1 "2024-09-07T07:53:52Z")
Do elasticsearch support Heterogeneous Cluster? Like if I have 1 master node at 7.17.6 , 2 data nodes at 7.17.6 & 1 data node at 8.11.1 will that work? This is to confirm that whether we can rolling upgrade ES without any downtime with both read and writes coming during the process. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 7, 2024, 9:09am UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/2 "2024-09-07T09:09:24Z")
You should always aim to have 3 master eligible nodes in a cluster as that allows the cluster to continue operating even if you lose one node. It also allows to rolling upgrades while keeping the cluster available. If you only have one master eligible node you should address this immediately. > [@Aditya\_Teltia](#): > > Like if I have 1 master node at 7.17.6 , 2 data nodes at 7.17.6 & 1 data node at 8.11.1 will that work? The first point I would make is that you should upgrade to the latest major version, not 8.11. You can do this node-by-node in a rolling fashion (at least as long as you have 3 master eligible nodes) but should never run with different versions in the cluster for any longer period of time, just during the rolling upgrade. ---
**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\ **Post date:** [September 7, 2024, 10:32am UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/3 "2024-09-07T10:32:19Z")
Is there any recommended order as well in which we should perform rolling upgrade? Like data nodes first then master nodes or vice versa? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 7, 2024, 10:49am UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/4 "2024-09-07T10:49:11Z")
I would recommend following [the instructions available in the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html). ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 7, 2024, 1:14pm UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/5 "2024-09-07T13:14:01Z")
> [@Aditya\_Teltia](#): > > This is to confirm that whether we can rolling upgrade ES without any downtime with both read and writes coming during the process. Keep in mind that with the infrastructure that you described you will have downtime, you have only one master, so when the master is down, your cluster will be down. You would need to add 2 more nodes to be master to be able to do a rolling upgrade without downtime. ---
**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\ **Post date:** [September 7, 2024, 1:44pm UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/6 "2024-09-07T13:44:33Z")
Yes I understood this, Although currently while trying the same locally I am facing an issue that is, I have 1 master node, 3 data nodes. I update one data node from 7.17.6 to 8.11.1. Everything works fine till here. Data node with ES8 is able to read all docs and perform search but as soon as I perform write, document gets written to other two data nodes that are on ES7, but the request keeps on running, and doc does not get written to ES8 node. ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/0/20cddc27f3364923ae036a9ccbb1e61f9d87dbbd.png) ```auto index shard prirep state docs store ip node .geoip_databases 0 p STARTED 34 32.4mb 127.0.0.1 data-1 .geoip_databases 0 r STARTED 34 32.4mb 127.0.0.1 data-2 syn 0 r STARTED 59223 129.8mb 127.0.0.1 data-1 syn 0 r STARTED 59221 129.8mb 127.0.0.1 data-2 syn 0 p STARTED 59223 129.8mb 127.0.0.1 data-3 ``` data-2 is on 8.11.1 version on which the document didn't get written. Cluster is green. ```auto { "cluster_name": "my-cluster", "status": "green", "timed_out": false, "number_of_nodes": 4, "number_of_data_nodes": 3, "active_primary_shards": 2, "active_shards": 5, "relocating_shards": 0, "initializing_shards": 0, "unassigned_shards": 0, "delayed_unassigned_shards": 0, "number_of_pending_tasks": 0, "number_of_in_flight_fetch": 0, "task_max_waiting_in_queue_millis": 0, "active_shards_percent_as_number": 100.0 } ``` I can see following logs in master-1 node ```auto [2024-09-07T18:56:29,478][INFO][o.e.c.m.MetadataMappingService] [master-1] [syn/z7FdISbDS72RueGOU2aFhw] update_mapping [universalMessage] [2024-09-07T18:59:47,220][WARN][o.e.c.InternalClusterInfoService] [master-1] failed to retrieve stats for node [NRMhqi_zSGCOi2qa0YmbDg]: [data-2][127.0.0.1:9304][cluster:monitor/nodes/stats[n]] request_id [4356] timed out after [15018ms] [2024-09-07T18:59:47,221][WARN][o.e.c.InternalClusterInfoService] [master-1] failed to retrieve shard stats from node [NRMhqi_zSGCOi2qa0YmbDg]: [data-2][127.0.0.1:9304][indices:monitor/stats[n]] request_id [4360] timed out after [15018ms] [2024-09-07T18:59:59,689][WARN][o.e.t.TransportService] [master-1] Received response for a request that has timed out, sent [27.3s/27383ms] ago, timed out [12.3s/12365ms] ago, action [indices:monitor/stats[n]], node [{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{fC4yJu4FSlKDg1JhwpLYcA}{127.0.0.1}{127.0.0.1:9304}{d}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true}], id [4360] [2024-09-07T18:59:59,690][WARN][o.e.t.TransportService] [master-1] Received response for a request that has timed out, sent [27.3s/27383ms] ago, timed out [12.3s/12365ms] ago, action [cluster:monitor/nodes/stats[n]], node [{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{fC4yJu4FSlKDg1JhwpLYcA}{127.0.0.1}{127.0.0.1:9304}{d}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true}], id [4356 ``` And following logs in node having primary shard that is data-3 node ```auto [2024-09-07T18:51:31,192][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-1] Released all partial updates permits [2024-09-07T18:51:31,193][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] finalizing recovery took [21.2ms] [2024-09-07T18:51:31,229][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-1] finalizing recovery took [37.2ms] [2024-09-07T18:55:15,231][INFO][o.e.t.ClusterConnectionManager] [data-3] transport connection to [{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{lBECEGbRQlSo0FTWenb0cQ}{127.0.0.1}{127.0.0.1:9304}{d}] closed by remote [2024-09-07T18:55:15,294][INFO][o.e.c.s.ClusterApplierService] [data-3] removed {{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{lBECEGbRQlSo0FTWenb0cQ}{127.0.0.1}{127.0.0.1:9304}{d}}, term: 1, version: 345, reason: ApplyCommitRequest{term=1, version=345, sourceNode={master-1}{Fmez2XfzTwCHboP1rGJj6g}{O53ok8QMR3CfeZLLBAYs4Q}{127.0.0.1}{127.0.0.1:9300}{m}{xpack.installed=true, transform.node=false}} [2024-09-07T18:55:45,000][INFO][o.e.c.s.ClusterApplierService] [data-3] added {{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{fC4yJu4FSlKDg1JhwpLYcA}{127.0.0.1}{127.0.0.1:9304}{d}}, term: 1, version: 346, reason: ApplyCommitRequest{term=1, version=346, sourceNode={master-1}{Fmez2XfzTwCHboP1rGJj6g}{O53ok8QMR3CfeZLLBAYs4Q}{127.0.0.1}{127.0.0.1:9300}{m}{xpack.installed=true, transform.node=false}} [2024-09-07T18:55:47,208][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] Acquiring all partial updates permits [2024-09-07T18:55:47,212][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] Acquired all partial updates permits [2024-09-07T18:55:47,218][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] recovery [phase2]: sending transaction log operations (from [9999] to [9998] [2024-09-07T18:55:47,226][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] recovery [phase2]: took [7.5ms] [2024-09-07T18:55:47,226][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] finalizing recovery [2024-09-07T18:55:47,226][INFO][o.e.i.s.ReplicationTracker] [data-3] [syn][0] markAllocationIdAsInSync Dsc_0TpCT_O1B8evpIwBJQ 9998 9998 9998 [2024-09-07T18:55:47,226][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] Released all partial updates permits [2024-09-07T18:55:47,244][INFO][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] finalizing recovery took [18ms] ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 7, 2024, 1:53pm UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/7 "2024-09-07T13:53:54Z")
There is no issue in what you shared, those are only INFO and WARN logs, no ERROR logs. Also, you have no unassigned shards and it also shows that the `syn` index has 1 primary shard and 2 replicas shards, so it was written in all your 3 data nodes. As mentioned, you should have different versions only temporarily, please update the rest of your cluster and come back if you have any issues. ---
**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\ **Post date:** [September 7, 2024, 2:58pm UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/8 "2024-09-07T14:58:34Z")
But \_cat/shards?v shows ```auto index shard prirep state docs store ip node .geoip_databases 0 p STARTED 34 32.4mb 127.0.0.1 data-1 .geoip_databases 0 r STARTED 34 32.4mb 127.0.0.1 data-2 syn 0 r STARTED 59223 129.8mb 127.0.0.1 data-1 syn 0 r STARTED 59221 129.8mb 127.0.0.1 data-2 syn 0 p STARTED 59223 129.8mb 127.0.0.1 data-3 ``` which shows that new document that I ingested got written to data-1 & data-3 node but wasn't written to data-2 node which I upgraded to ES8. And also `localhost:9200/syn/_doc/1` request keeps on running and doesn't give any response. Not sure but this seems like an issue. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 7, 2024, 3:04pm UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/9 "2024-09-07T15:04:03Z")
Data is not written to only the upgraded node, it is written to the cluster as a whole and will go to the nodes that hold the appropriate indices/shards. ---
**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\ **Post date:** [September 8, 2024, 5:42am UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/10 "2024-09-08T05:42:48Z")
I get it and exactly why I am asking it didn't got written to data-2 node since it also has a replica of `syn` index to which I ingested document as you can see in `_cat/shards` response. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 8, 2024, 5:57am UTC](https://discuss.elastic.co/t/heterogeneous-cluster/366184/11 "2024-09-08T05:57:03Z")
It sounds like you did not follow through on the recommendation to complete the upgrade and not leave the cluster with different versions in it. Shards that are allocated to the newer node will be associated with that version and it can not be read by older nodes. If you end up with a primary on the newer node it can not be replicated to older nodes if you need to rebalance or recover as they can not read the newer format. I am not sure whether this also applies to normal replication. This is why you do not want to run clusters with any version discrepancies for any period of time.