I've got a general question about datamanagement in elastic - we're running a couple of on-premise clusters and the data we produce are mostly from textfiles or are generated in various systems, collected in Redis from were we are transferring them to Elasticsearch an parallel we generate backup text files just in case anything goes wrong or has to be replayed somewhere else.
So my question, isn't it often (i don't say always) more convenient to backup the raw data textfiles so they can imported again if needed since there aren't any problems with version upgrades or as we had it migration issues from a 5.2 cluster to a 8.x cluster ... with the restore method the data has to be redindex anyway (if i got this right) and when we have the raw textfiles we can to the same ... it's just something i was wondering since we are keeping the textfiles anyway.
How long does it take to restore the original data from file as opposed to a restore from snapshot?
Both have their benefits, but which is easier for you?
It depends how much we want to restore - 2 weeks can be imported within hours - 2 month would definitly longer. The thing is, we had a lot of troubles with our repositories when upgrading from a 5.x cluster to a higher version, where the snapshot could not be imported any longer - so we had to import the snapshots - upgrade, upgrade and than we hat the opportunity to create a new repo, snapshot again etc. ... this was so cumbersome that we really switched to re-import the data which was needed like an "on demand import". We still use snapshot/restore - it's a great feature but we additionally keep the raw data so we can decide what to do in which case ... i was just wondering how others are handling datastreams and especially how to archive data over years without an LCM of hot-warm-cold architecture.
Yep that's understandable and we've done a tonne of work, with more to come afaik, to make that easier.
Thank you for your response!
Definitly there was much improvement in this area and i'm looking forward to upcoming changes! 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.