I'm upgrading the Elasticsearch cluster, currently I have 10 node clusters running on individual instances , All nodes are having older version, if I upgrade one by one all nodes starting from data node to master node. Will there be anything for which I need to cautious about before proceeding.
I'm doing this via terraform, firstly upgrading the 10th data node by transferring all data to other instances and making it empty and then deleting that node and replacing with the new node which will be having the latest Elasticsearch version.
The links Dinesh sent are your first point of call. One gives you all the links to breaking changes and the other shows how to do a rolling upgrade.
You upgrade strategy does seem mildly strange. Why not do a rolling upgrade and keep the data where it is?
Rolling upgrades work very well especially with RPMS. I have done over four in the last few months. They are really are robust. However all of this needs to be tested. Things change from version to version and you may well need to make yml changes etc. You will also need a new Kibana if you are using it.
A cluster upgrade isn't just upgrading the cluster. Things feed into it e.g. Logstash, Kibana, Beats etc. Check the upgrade compatibility matrix to check which parts of your environment needs to be upgraded. You may find that not all of it will need upgrading and can be done later. Also be wise to OS versions.
When you run your upgrade, I would start with a master. You should have three dedicated masters meaning that it the upgrade were to fail on the first one, the cluster would still work and you would have time to resolve the issue before moving on. If a data not went, the that may cause more issues.
Sounds like you really need to think it through and test it thoroughly.
The problem is I dont have test environment and setting up test env will be another big task. Do I need to upgrade logstash, kibana and filebeat as well with Elasticsearch version upgrade?
What if I only upgrade the cluster?
Without a test environment, your upgrade is likely not to succeed first time. There are many breaking changes between releases. Most of these won't affect you but some will. You need to test them to ensure they work before upgrading. Otherwise you will just get into a mess.
As to what needs upgrading, along with Elasticsearch, can be found in the compatibility matrix. Logstash is generally pretty good. Version 6.8.x will work up to 7.17.x. Likewise with Beats. However always check the small print. Kibana on the other hand does need to be on the same version as Elasticsearch. The point I am making is that you need to read the documentation, but if you are smart, you can bring up your system to a mature version e.g. 7.17.x and not have to change everything at once. You don't have to rush to 8.0. We run Logstash 6.8 and Filebeat 6.8 with 7.16.1. As a result when I upgraded to 7.16.1, I didn't need to change the whole stack. It gave me more time to plan the rest which I am doing now.
Read the documentation, plan and test. It is far more likely to succeed.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.