ES rolling upgrade from 6.3.2 to 6.8.22, ES plug-in version should be upgraded to 6.8.22?

[qiuxb (qiuxinbo)]

In order to fix the log4j2 vulnerability, we plan to upgrade the ES version from 6.3.2 to 6.8.22, and 7.x to 7.16.2. When reading the rolling upgrade document, the rolling upgrade document for 7.16.2 mentioned that the plugin needs to be upgraded. version, but within the rolling upgrade documentation for 6.8.22, there is nothing about upgrading plugins.
I have some questions:

1. Do I need to upgrade the plug-in version in time during the rolling upgrade process?
2. Is the plugin version backward compatible? Will the new version of the plugin be deprecated?
3. In a cluster, some ES nodes are upgraded to 6.8.22 or 7.16.2, and the plug-in version is upgraded to 6.8.22 or 7.16.2. If multiple plug-in versions coexist, will it affect the normal access of the cluster?

[warkolm (Mark Walkom)]

1. Yes, do it before you restart the node
2. Which plugin?
3. Yes, Elasticsearch likely won't start

[qiuxb (qiuxinbo)]

1. Yes, do it before you restart the node
   --After shutting down the node and before starting the node with ES 6.8.22 binaries, use elasticsearch-plugins install plugin-name from the 6.8.22 package ?

2. Which plugin?

• analysis-ik , version : 6.3.2 , description : IK Analyzer for Elasticsearch }
• analysis-pinyin , version : 6.3.2 , description : Pinyin Analysis for Elasticsearch }
• analysis-ik , version : 6.3.2 , description : IK Analyzer for Elasticsearch }
• analysis-pinyin , version : 6.3.2 , description : Pinyin Analysis for Elasticsearch }
• analysis-ik , version : 7.4.2 , description : IK Analyzer for Elasticsearch }
• analysis-pinyin , version : 7.4.2 , description : Pinyin Analysis for Elasticsearch }
• analysis-jieba , version : 6.4.0 , description : A jieba analysis of plugins for Elasticsearch }
• repository-hdfs , version : 6.8.22 , description : The HDFS repository plugin adds support for Hadoop
• sql , version : 6.3.2.0 , description : Query elasticsearch using SQL }
• analysis-jieba , version : 6.0.0 , description : A jieba analysis of plugins for Elasticsearch }
• bitmap-plugin , version : 6.3.2 , description : The bitmap-engine plugin allows to compute bitmap of a field s values at search-time}
• analysis-stconvert , version : 6.3.2 , description : STConvert is a analysis plugin that convert Chinese
• analysis-ik , version : 6.4.1 , description : IK Analyzer for Elasticsearch }
• analysis-jieba , version : 6.4.1 , description : A jieba analysis of plugins for Elasticsearch }
• analysis-ik , version : 7.2.0 , description : IK Analyzer for Elasticsearch }
• analysis-icu , version : 6.3.2 , description : The ICU Analysis plugin integrates Lucene ICU module
• analysis-pinyin , version : 7.2.0 , description : Pinyin Analysis for Elasticsearch }
• sql , version : 7.2.0.0 , description : Query elasticsearch using SQL }
• analysis-jieba , version : 6.3.2 , description : A jieba analysis of plugins for Elasticsearch }
• analysis-pinyin , version : 6.4.1 , description : Pinyin Analysis for Elasticsearch }
• analysis-remote , version : 1.0.0 , description : Remote Analysis plugin for Elasticsearch }
• feature-log , version : 6.3.2 , description : feature log plugin for Elasticsearch }
• analysis-remote , version : 1.0.0 , description : Remote Analysis plugin for Elasticsearch }

3. Yes, Elasticsearch likely won't start

I use the new 6.8.22 zip package to unzip it to a new directory, copy the old elasticsearch.yaml and jvm.options files to overwrite the config in the new directory, and start the node after that. The node starts normally without failure. And successfully joined the cluster, but there is no return when executing the elasticsearch-plugin list command, the same as using cat/plugins

[qiuxb (qiuxinbo)]

ES is upgraded from version 6.3.2 to 6.8.22 in a rolling manner. After the upgrade, the cluster status is abnormal. How to roll back to 6.3.2?

Thanks for the reply, I have a question about rolling upgrades, the part marked in the following text, is it understood that it is not allowed to accept multiple ES versions for a long time during the upgrade? However, the actual rolling upgrade requires a gradual upgrade of nodes one by one, and needs to wait for the node status to recover and the cluster status to recover to green, which may take a long time. During this period, two versions of ES must exist in the cluster at the same time. R…

I still have a question, what are the main factors that affect the coexistence of multiple versions of ES during ES rolling upgrades? Why is it safer to be under the hour?

[warkolm (Mark Walkom)]

If you end up with an index that has shards of different lucene versions, it can cause issues.

[qiuxb (qiuxinbo)]

Is it possible to do a pre-check to find out earlier? How can I check that different indexes contain shards of different lucene versions?

[warkolm (Mark Walkom)]

They won't (or shouldn't), because a refresh happens at an index level and that is how the versions usually change.

[qiuxb (qiuxinbo)]

At present, many of our indexes occupy terabytes of space, and it may take 1-2 hours to restart the ES node to restore the green state, so I am worried that the coexistence time of multiple versions of ES will reach several days or weeks. Will there be more the problem arises?