Question regarding the upgrade of an Elasticsearch cluster, which has over 50 nodes, to a new cluster running RedHat
operating system.
The question is: should we build the new cluster in a Docker environment or a Kubernetes environment?
What would you recommend?
Which of those 2 technologies do you and your team know best, and have most (real world) experience using? I'd recommend using that one.
For me, that would be k8s, but I am not you or your team.
If you dont know much about either, I'd question why you are limiting yourself to those choices.
btw, I'm not quite getting the mention of "RedHat operating system ", for which I presume you mean RHEL? Personally speaking, if you are coming from some other linux distro, say Debian or Ubuntu or whatever, I don't think that you are going to RHEL plays much of a role here. Maybe there is something I am not considering.
I presume also you will be migrating, rather than actually upgrading, though some might think that is just semantics.
You're right, we're definitely talking about a migration, not an upgrade.
If we donโt limit ourselves to these two options, what other recommended alternatives remain?
is this a cloud provider installation (on AWS, azure, ...). If on AWS, just because I've done it before, I'd use k8s.
Are you a paying customer? For a significant installation I'd certainly consider using Elastic Cloud, obviously this has associated costs but it also saves, just (sometimes) in less obvious ways.
I'm also old school, maybe just old, so if someone is going me 50 servers + storage and a data center and asking for a 50-node cluster, I'm probably just building it onto those servers directly. Why use another layer at all? Especially if I've no experience with docker or k8s, which was my first question.
On other hands if someone is saying "build a 50-node cluster ontop of this existing VMware/k8s/whatever" then obviously thats the infrastructure I'm working with.
btw, though its not directly related to the docker/k8s/... question, you didn't mention what version you are migrating from/to. There was thread last week when someone was migrating from 1.x to 8.x (sic). Relevance is only to ensure you understand the various changes between versions (major and minor) that might impact your overall solution.
We are running elasticsearch version 7.5.1
OK, if sticking with that release, which btw is 5+ years old, then IMO it does not really impact choice of docker/k8s/whatever, at least not that I know.
If you are going to upgrade, you need go via 7.17.x, cannot go directly to 8.x. And heaps of changes in stuff over those 5+ years is massive, so ... IMO thats a whole different project, which you are discussing in a different thread.
For what it's worth, I would also separate the migration from the upgrade. not try to do both at once. Because I am extremely cautious.
I think I've added as much as I can, so good luck with the migration, whatever route you decide to take.
Our cluster servers are virtual servers on a VMWARE platform.
I still dont really see the significant advantage of a docker / k8s layer for an elasticsearch cluster, if you are anyways using VMware and have a bunch of VMs running (I assume) RHEL.
As you'll know already, the key features are performant IO and in many use cases plenty of RAM.
Long time ago some (architects/developers/managers/...) where I worked thought - we have this great on-prem openshift cluster, lots of CPU and RAM, lots of ceph storage, lets put our ELK (and couchbase and mongo and redis) clusters on there too. As was predicted, as soon as significant load appeared, i.e. contact with real world/users, a nice looking block diagram fell apart rapidly. Ouch. elasticsearch itself was shortly thereafter moved to a few dedicated VMs and lived happily ever after (til it all got migrated to AWS). But, as I write here often, I'm old school, so would be interesting if the New Kids on the Block chipped in too.
Thanks Kewin,
New Kids on the Block are invited to chip in too.
What about Elasticsearch cluster High Availability? Are there any advantages to Docker/Kubernetes compared to VMware?
ยฉ 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.