Hello,
From what I understand, the snapshots are full (similar to full backups in SQL Server) and there is no option of incremental snapshots.
- Elasticsearch snapshots are increments, which I will explain in more detail.
- You can restore an index to any point in time at which you have taken a snapshot.
- If you took a snapshot at 8 PM yesterday and have a catastrophic failure sometime after without having taken another snapshot, you can restore exactly what you had at 8 PM yesterday. (I can't imagine how it could be otherwise.)
Please have a look at the documentation.
Snapshots are taken incrementally. This means that when it creates a snapshot of an index, Elasticsearch avoids copying any data that is already stored in the repository as part of an earlier snapshot of the same index. Therefore it can be efficient to take snapshots of your cluster quite frequently.
The operational outcome of your snapshot command on an index is that all of the segment files for that index in your cluster have copies in your snapshot repository, and that the metadata in the repository has references that say "snapshot sn_name of index i_name comprises segments list of segments."
The repository will never contain more than one copy of a segment - if you take a quick series of snapshots on an index that isn't currently receiving any docs to index, you will probably have no additional segment data transferred*, just additional metadata references created.
Similarly, when you restore a named snapshot of an index, the outcome is that the population of segments for that index in the cluster will be the segments listed in the metadata of the repository for that snapshot of that index. Any segments of which there is already a copy in the cluster won't need to be streamed in. Any segments that exist in the cluster for that index but weren't part of that snapshot will be removed.
Because of this, you can take snapshots quite frequently without the concern you've alluded to about "full backups", and precisely mitigate the risk you've mentioned of losing large amounts of data due to a long interval between backups.
One caution I will provide is that if you have a scenario where you are indexing heavily into a daily index, and taking frequent snapshots, you will have, in a sense, a lot of redundant data in your repository. That is, the same documents will be held in numerous segment files, because the segment files are constantly being merged in such an index.
This is fine, but unless you have a need to be able to restore a perfectly healthy index to lots of arbitrary points in time, all you are really interested in is the last known good state. Hence, you can apply a sanitation process where you delete older named snapshots. During the process of deleting named snapshots, when the system sees that the last metadata reference to a particular segment in the repository is removed, then that segment can also be removed.
*The exception would be when segment merging happens See Merge for more info.