Partly downgrade 5.5.2 to 5.1.2 possible?

[asp]

Hi,

I am planing to upgrade the ELK stack from 5.1.2 to 5.5.2.
We are using daily index rotating.

Is anything happening with the old indices, which have been created with kibana 5.1.2 and where no new data has been added? Are they modified in any way or is elasticsearch them just reading in a backward compatible mode?

My thoughts are going in following direction:
If anything critical encounters after migration, I need to rollback in some kind.

Option 1:
Doing a snapshot of the vm and revert to the snapshot, if anything happens.
Then the data between migration and revert timestamp will be lost / needs to be reprocessed.
- pro: easy to do
- con: I cannot do it myself, a lot of process overhead in my company, additional money for the snapshot as long as it runs.

Option 2:
Following scenery seems to work on the first glimpse in my local developer vm:

• Before migration I will snapshot the .kibana index and all indices of the current day while still 5.1.2 is running
• migration, insert new data for today, then decission to rollback
• stop elasticsearch, logstash, kibana
• change the to the old binaries of ELK 5.1.2 to be started
• close indices of .kibana and current day
• restore snapshot made before migration
  - pro: I can do it myself (-> more flexibility), probably less costs in our company
  - con: more complicated, not knowing if it works in real life.

So for making my decisson I need your information, if Option 2 will work - if 5.5.2 will modify any old indexes which makes them not readable for version 5.1.2.

For option 3 (exporting all indices before starting my migration I do not have enough storage available.
Thanks, Andreas

[warkolm (Mark Walkom)]

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats feels left out

If you upgrade and an index is merged after that upgrade process, you cannot downgrade.
What sort of problems do you think you will run into that means you need to roll back?

[asp]

what do you mean with an "index is merged" ? When new data inserted to it?
If so, can I define an index as read-only so that a merge is not possible?

I am thinking of following possible problems:

• performance on productive environment is lower with 5.5.2 than with 5.1.2
• stability issues on productive environment
• something critical which slipped through our tests (some dashboard will not work (I already encountered an issue with excludes in visualizations when upgraded from 5.1.2 to 5.2.0. There we'd needed to rollback, if we had not seen it in testing phase - so I am just a bit careful)

Unfortunately in my project kibana is just a kind of a one man show although it has become quite critical to us. And I don't have an adequate testing environment to do loadtests because of few budget. That's why I want to have a rollback plan in my pocket.

[warkolm (Mark Walkom)]

@asp

what do you mean with an “index is merged” ? When new data inserted to it?

Potentially, or if more shards are created and the system needs to merge them.

@asp

If so, can I define an index as read-only so that a merge is not possible?

Yes, but that won't stop indices that are being written to now from being merged.

@asp

Unfortunately in my project kibana is just a kind of a one man show although it has become quite critical to us. And I don’t have an adequate testing environment to do loadtests because of few budget.

Basically you don't have many options here due to the lack of importance placed on how critical the system is.

You can take VM snapshots, but Elasticsearch won't really like that. You can't take backups from Elasticsearch directly. You can't create a test system.
Can you spin up a new cluster?

[asp]

we have only one vm, running an single instance
I could do a second / parallel installation on the same server, but I think that processing two instances in parallel, the resources might not be enough.

Is there

@warkolm

what do you mean with an “index is merged” ? When new data inserted to it?

Potentially, or if more shards are created and the system needs to merge them.

is there a way to disable that behavior temporary? I think 1 or 2 days are enough to see if everything works as expected.

vm snapshot is planed to be done when the system is down. So it should be transparent to es. Of course it may affect the performance negatively.

[warkolm (Mark Walkom)]

@asp

is there a way to disable that behavior temporary? I think 1 or 2 days are enough to see if everything works as expected.

No there's not, it's not anything we'd recommend anyway. And besides, any new index created will only be readable on the current (upgraded) version.

@asp

vm snapshot is planed to be done when the system is down. So it should be transparent to es. Of course it may affect the performance negatively.

That's likely the easiest way then.

[asp]

ok, thanks a lot then.
We will discuss in my team, wich risk we are willing to take and which way to go.