I am planing to upgrade the ELK stack from 5.1.2 to 5.5.2.
We are using daily index rotating.
Is anything happening with the old indices, which have been created with kibana 5.1.2 and where no new data has been added? Are they modified in any way or is elasticsearch them just reading in a backward compatible mode?
My thoughts are going in following direction:
If anything critical encounters after migration, I need to rollback in some kind.
Option 1:
Doing a snapshot of the vm and revert to the snapshot, if anything happens.
Then the data between migration and revert timestamp will be lost / needs to be reprocessed. - pro: easy to do - con: I cannot do it myself, a lot of process overhead in my company, additional money for the snapshot as long as it runs.
Option 2:
Following scenery seems to work on the first glimpse in my local developer vm:
Before migration I will snapshot the .kibana index and all indices of the current day while still 5.1.2 is running
migration, insert new data for today, then decission to rollback
stop elasticsearch, logstash, kibana
change the to the old binaries of ELK 5.1.2 to be started
close indices of .kibana and current day
restore snapshot made before migration - pro: I can do it myself (-> more flexibility), probably less costs in our company - con: more complicated, not knowing if it works in real life.
So for making my decisson I need your information, if Option 2 will work - if 5.5.2 will modify any old indexes which makes them not readable for version 5.1.2.
For option 3 (exporting all indices before starting my migration I do not have enough storage available.
Thanks, Andreas
FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats feels left out
If you upgrade and an index is merged after that upgrade process, you cannot downgrade.
What sort of problems do you think you will run into that means you need to roll back?
what do you mean with an "index is merged" ? When new data inserted to it?
If so, can I define an index as read-only so that a merge is not possible?
I am thinking of following possible problems:
performance on productive environment is lower with 5.5.2 than with 5.1.2
stability issues on productive environment
something critical which slipped through our tests (some dashboard will not work (I already encountered an issue with excludes in visualizations when upgraded from 5.1.2 to 5.2.0. There we'd needed to rollback, if we had not seen it in testing phase - so I am just a bit careful)
Unfortunately in my project kibana is just a kind of a one man show although it has become quite critical to us. And I don't have an adequate testing environment to do loadtests because of few budget. That's why I want to have a rollback plan in my pocket.
Potentially, or if more shards are created and the system needs to merge them.
Yes, but that won't stop indices that are being written to now from being merged.
Basically you don't have many options here due to the lack of importance placed on how critical the system is.
You can take VM snapshots, but Elasticsearch won't really like that. You can't take backups from Elasticsearch directly. You can't create a test system.
Can you spin up a new cluster?
we have only one vm, running an single instance
I could do a second / parallel installation on the same server, but I think that processing two instances in parallel, the resources might not be enough.
Is there
is there a way to disable that behavior temporary? I think 1 or 2 days are enough to see if everything works as expected.
vm snapshot is planed to be done when the system is down. So it should be transparent to es. Of course it may affect the performance negatively.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
