We use the hdfs as our elasticsearch repositories to back up and restore.We have alread installed the 'elasticsearch-repository-hdfs' plugin in all the elasticsearch nodes.
I have a question as followed:
As the backup progress is incremental, if the snapshot files in the hdfs were compressed, such as in 'har' format, can we still continuing backing up new snapshots?
If not, how can we do back up work on conditions that the existing snapshots were compressed?
Snapshot/Restore works only if the underlying storage is not touched/affected. Moving the files, compressing them, etc.. obviously changes their content and at that point ES cannot use them any more.
Further more since the snapshots are incremental it means each snapshot depends on the previous one - when you delete one, ES will make the right choice and will delete the data that is no longer needed. However this again is something that ES does already and any interference with it will screw things up.
Now you could manually manage the snapshots as in create a repository, take some snapshots, then close it, move the files/compress them but it means in case of a snapshot/restore, you have to do that backwards as otherwise ES will not do an incremental backup but rather a full one.
Note that since ES only backups the delta, new backups are as cheap as they can be.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.