Snapshot - Correct order to manage them


(Roger) #1

hi all,
I would like to understand the correct way to work with snapshots.
This is my scenario:

  • NFS space disk: 100 GB
  • First snapshot created: 94 GB.
  • After that I changed more or less 10 GB of documents
  • Tried to create another snapshot.

It was started correctly but after a while it crashed since there was not free space left.
So my question is: as best practice, should I delete snapshot before to create a new one?
is there a way to check the free space on shared disk to understand if will be possible to complete the procedure?


(David Turner) #2

As a best practice, no. Each snapshot will re-use any parts of previous snapshots that it can, so keeping old snapshots around should help make subsequent snapshots faster, and is normally a good idea. It's also a good idea to have a bit more leeway in snapshot storage than in the situation you describe. There's no practical way to work out how much extra space a snapshot may take - even if you add no more documents, things like background merges may create new segments and add to the size of the repository. In the worst case you might have to copy the entire index again, so the amount of free space needed to be sure the snapshot will succeed is the size of the index.


(Roger) #3

hmmm....so, in my case, there is no solution.
my biggest problem is that the second snapshot will start anyway and if there will not free space left on disk I will also not be able to remove snapshot using API commands. so the only one solution at the end will be to remove everything from the disk and start again with a snapshot :open_mouth:


(David Turner) #4

That's not quite what I said: the best solution is to obtain more disk space.

If this is impossible for some reason then I think you will have to delete each snapshot before creating the next one. I wouldn't call it a best practice, but it is at least a solution. You will have to be careful that your data doesn't grow by more than a few percent or else you won't be able to make any snapshots at all. You will also have to be comfortable with no snapshot for a period of time.

This sounds related to https://github.com/elastic/elasticsearch/issues/13159 which will be solved in due course.


(Roger) #5

Could be an idea to waste with rubbish files some megabytes on file system? In this way, if the snapshot fails because there is no space left on disk, I will be able to delete my rubbish files and still manage snapshot using API.
Is this really stupid, I know...


(David Turner) #6

Hmm. It might work, but I am unsure of what state the repository would be in after hitting an out-of-space error so I'm not 100% sure it'll always behave as you might expect. I just tried this and was indeed able to delete snapshots after removing the waste-of-space files, but it did seem to leave some unexpected things behind. I think starting with a fresh repository would be safer.