Do elasticsearch support Heterogeneous Cluster?
Like if I have 1 master node at 7.17.6 , 2 data nodes at 7.17.6 & 1 data node at 8.11.1 will that work?
This is to confirm that whether we can rolling upgrade ES without any downtime with both read and writes coming during the process.
You should always aim to have 3 master eligible nodes in a cluster as that allows the cluster to continue operating even if you lose one node. It also allows to rolling upgrades while keeping the cluster available. If you only have one master eligible node you should address this immediately.
The first point I would make is that you should upgrade to the latest major version, not 8.11. You can do this node-by-node in a rolling fashion (at least as long as you have 3 master eligible nodes) but should never run with different versions in the cluster for any longer period of time, just during the rolling upgrade.
Is there any recommended order as well in which we should perform rolling upgrade? Like data nodes first then master nodes or vice versa?
I would recommend following the instructions available in the documentation.
Keep in mind that with the infrastructure that you described you will have downtime, you have only one master, so when the master is down, your cluster will be down.
You would need to add 2 more nodes to be master to be able to do a rolling upgrade without downtime.
Yes I understood this, Although currently while trying the same locally I am facing an issue that is,
I have 1 master node, 3 data nodes. I update one data node from 7.17.6 to 8.11.1. Everything works fine till here. Data node with ES8 is able to read all docs and perform search but as soon as I perform write, document gets written to other two data nodes that are on ES7, but the request keeps on running, and doc does not get written to ES8 node.
index shard prirep state docs store ip node
.geoip_databases 0 p STARTED 34 32.4mb 127.0.0.1 data-1
.geoip_databases 0 r STARTED 34 32.4mb 127.0.0.1 data-2
syn 0 r STARTED 59223 129.8mb 127.0.0.1 data-1
syn 0 r STARTED 59221 129.8mb 127.0.0.1 data-2
syn 0 p STARTED 59223 129.8mb 127.0.0.1 data-3
data-2 is on 8.11.1 version on which the document didn't get written.
Cluster is green.
{
"cluster_name": "my-cluster",
"status": "green",
"timed_out": false,
"number_of_nodes": 4,
"number_of_data_nodes": 3,
"active_primary_shards": 2,
"active_shards": 5,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}
I can see following logs in master-1 node
[2024-09-07T18:56:29,478][INFO ][o.e.c.m.MetadataMappingService] [master-1] [syn/z7FdISbDS72RueGOU2aFhw] update_mapping [universalMessage]
[2024-09-07T18:59:47,220][WARN ][o.e.c.InternalClusterInfoService] [master-1] failed to retrieve stats for node [NRMhqi_zSGCOi2qa0YmbDg]: [data-2][127.0.0.1:9304][cluster:monitor/nodes/stats[n]] request_id [4356] timed out after [15018ms]
[2024-09-07T18:59:47,221][WARN ][o.e.c.InternalClusterInfoService] [master-1] failed to retrieve shard stats from node [NRMhqi_zSGCOi2qa0YmbDg]: [data-2][127.0.0.1:9304][indices:monitor/stats[n]] request_id [4360] timed out after [15018ms]
[2024-09-07T18:59:59,689][WARN ][o.e.t.TransportService ] [master-1] Received response for a request that has timed out, sent [27.3s/27383ms] ago, timed out [12.3s/12365ms] ago, action [indices:monitor/stats[n]], node [{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{fC4yJu4FSlKDg1JhwpLYcA}{127.0.0.1}{127.0.0.1:9304}{d}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true}], id [4360]
[2024-09-07T18:59:59,690][WARN ][o.e.t.TransportService ] [master-1] Received response for a request that has timed out, sent [27.3s/27383ms] ago, timed out [12.3s/12365ms] ago, action [cluster:monitor/nodes/stats[n]], node [{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{fC4yJu4FSlKDg1JhwpLYcA}{127.0.0.1}{127.0.0.1:9304}{d}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true}], id [4356
And following logs in node having primary shard that is data-3 node
[2024-09-07T18:51:31,192][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-1] Released all partial updates permits
[2024-09-07T18:51:31,193][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] finalizing recovery took [21.2ms]
[2024-09-07T18:51:31,229][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-1] finalizing recovery took [37.2ms]
[2024-09-07T18:55:15,231][INFO ][o.e.t.ClusterConnectionManager] [data-3] transport connection to [{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{lBECEGbRQlSo0FTWenb0cQ}{127.0.0.1}{127.0.0.1:9304}{d}] closed by remote
[2024-09-07T18:55:15,294][INFO ][o.e.c.s.ClusterApplierService] [data-3] removed {{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{lBECEGbRQlSo0FTWenb0cQ}{127.0.0.1}{127.0.0.1:9304}{d}}, term: 1, version: 345, reason: ApplyCommitRequest{term=1, version=345, sourceNode={master-1}{Fmez2XfzTwCHboP1rGJj6g}{O53ok8QMR3CfeZLLBAYs4Q}{127.0.0.1}{127.0.0.1:9300}{m}{xpack.installed=true, transform.node=false}}
[2024-09-07T18:55:45,000][INFO ][o.e.c.s.ClusterApplierService] [data-3] added {{data-2}{NRMhqi_zSGCOi2qa0YmbDg}{fC4yJu4FSlKDg1JhwpLYcA}{127.0.0.1}{127.0.0.1:9304}{d}}, term: 1, version: 346, reason: ApplyCommitRequest{term=1, version=346, sourceNode={master-1}{Fmez2XfzTwCHboP1rGJj6g}{O53ok8QMR3CfeZLLBAYs4Q}{127.0.0.1}{127.0.0.1:9300}{m}{xpack.installed=true, transform.node=false}}
[2024-09-07T18:55:47,208][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] Acquiring all partial updates permits
[2024-09-07T18:55:47,212][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] Acquired all partial updates permits
[2024-09-07T18:55:47,218][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] recovery [phase2]: sending transaction log operations (from [9999] to [9998]
[2024-09-07T18:55:47,226][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] recovery [phase2]: took [7.5ms]
[2024-09-07T18:55:47,226][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] finalizing recovery
[2024-09-07T18:55:47,226][INFO ][o.e.i.s.ReplicationTracker] [data-3] [syn][0] markAllocationIdAsInSync Dsc_0TpCT_O1B8evpIwBJQ 9998 9998 9998
[2024-09-07T18:55:47,226][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] Released all partial updates permits
[2024-09-07T18:55:47,244][INFO ][o.e.i.r.RecoverySourceHandler] [data-3] [syn][0][recover to data-2] finalizing recovery took [18ms]
There is no issue in what you shared, those are only INFO and WARN logs, no ERROR logs.
Also, you have no unassigned shards and it also shows that the syn index has 1 primary shard and 2 replicas shards, so it was written in all your 3 data nodes.
As mentioned, you should have different versions only temporarily, please update the rest of your cluster and come back if you have any issues.
But _cat/shards?v shows
index shard prirep state docs store ip node
.geoip_databases 0 p STARTED 34 32.4mb 127.0.0.1 data-1
.geoip_databases 0 r STARTED 34 32.4mb 127.0.0.1 data-2
syn 0 r STARTED 59223 129.8mb 127.0.0.1 data-1
syn 0 r STARTED 59221 129.8mb 127.0.0.1 data-2
syn 0 p STARTED 59223 129.8mb 127.0.0.1 data-3
which shows that new document that I ingested got written to data-1 & data-3 node but wasn't written to data-2 node which I upgraded to ES8.
And also localhost:9200/syn/_doc/1 request keeps on running and doesn't give any response. Not sure but this seems like an issue.
Data is not written to only the upgraded node, it is written to the cluster as a whole and will go to the nodes that hold the appropriate indices/shards.
I get it and exactly why I am asking it didn't got written to data-2 node since it also has a replica of syn index to which I ingested document as you can see in _cat/shards response.
It sounds like you did not follow through on the recommendation to complete the upgrade and not leave the cluster with different versions in it.
Shards that are allocated to the newer node will be associated with that version and it can not be read by older nodes. If you end up with a primary on the newer node it can not be replicated to older nodes if you need to rebalance or recover as they can not read the newer format. I am not sure whether this also applies to normal replication. This is why you do not want to run clusters with any version discrepancies for any period of time.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.