Snapshot files are created every time even no change in index


(shoebalig) #1

I have a cron-job runs daily. There are chances that there is no modification or changes found in original index. however cron-job create snapshot file again on its scheduled time. Due to which number of snapshot files is too large. which leads restoration of index, takes a lot of time.
Is there anyway I should create snapshot file only when there is a difference in index data?
What if I retain only last 5 or last 10 snapshot? Will this cause any data lose anyway? And on restore whole index will be restored?

Please suggest.


(Mark Walkom) #2

Snapshots are taken on the shard level, so if a shard changes due to things
like merging, then it needs to record this.
You can delete snapshots, ES will just hold onto the shards it needs.

PS - We're moving to https://discuss.elastic.co/, please join us there for
any future discussions!

On 28 May 2015 at 21:13, shoebalig shoebalig@gmail.com wrote:

I have a cron-job runs daily. There are chances that there is no
modification or changes found in original index. however cron-job create
snapshot file again on its scheduled time. Due to which number of snapshot
files is too large. which leads restoration of index, takes a lot of time.
Is there anyway I should create snapshot file only when there is a
difference in index data?
What if I retain only last 5 or last 10 snapshot? Will this cause any data
lose anyway? And on restore whole index will be restored?

Please suggest.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Snapshot-files-are-created-every-time-even-no-change-in-index-tp4074965.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1432811614768-4074965.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9kFwpxBaDZt0C_HzUG_U5zHpTWgGBqfReEzrB0WK1umg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Allan Mitchell) #3

Mark

This seems counter intuitive. Incremental to me and technologies with which
I have worked before means that in order to restore to a known state you
need A+B+C..... In temporal order. Consider this example

DELETE /test

POST /test
{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"mytype" : {
"properties" : {
"myfield" : { "type" : "string","analyzer": "standard" }
}
}
}
}

POST /test/mytype/1
{
"myfield": "Doc 1"
}

Snapshot to snapshot1

POST /test/mytype/2
{
"myfield": "Doc 2"
}

Snapshot to snapshot2

POST /test/mytype/3
{
"myfield": "Doc 3"
}

Snapshot to snapshot3

Delete the test index in ES

Now go ahead and delete snapshot2

Restore the snapshot3 backup

POST /_snapshot/FILE_SYSTEM_ES/snapshot3/_restore

All 3 documents are in the index

Delete the index in ES again

Delete snapshot1
delete the index in ES
Restore the snapshot3 backup

All 3 documents are in the index

Even if the snapshot only takes the incremental changes it would seem the
previous snapshot is merged with the latest allowing the deletions you see
above. To get to my latest position I only need the latest snapshot. If I
want to get back to a previous state then I only need the relevant snapshot.

I dare say this is just nomenclature but incremental to me does not do what
the incremental in ES portrays

SQL Server Example.

If I take a FULL backup on Sunday and Incrementals on each night of the
week. If I delete everything before last night and try to restore only the
latest incremental I would be stuck.

If I delete snapshots in reverse order then the document count in the index
decreases as expected.

I cannot find any documentation anywhere on this process.

Allan

On 28 May 2015, at 23:07, Mark Walkom markwalkom@gmail.com wrote:

Snapshots are taken on the shard level, so if a shard changes due to things
like merging, then it needs to record this.
You can delete snapshots, ES will just hold onto the shards it needs.

PS - We're moving to https://discuss.elastic.co/, please join us there for
any future discussions!

On 28 May 2015 at 21:13, shoebalig shoebalig@gmail.com wrote:

I have a cron-job runs daily. There are chances that there is no
modification or changes found in original index. however cron-job create
snapshot file again on its scheduled time. Due to which number of snapshot
files is too large. which leads restoration of index, takes a lot of time.
Is there anyway I should create snapshot file only when there is a
difference in index data?
What if I retain only last 5 or last 10 snapshot? Will this cause any data
lose anyway? And on restore whole index will be restored?

Please suggest.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Snapshot-files-are-created-every-time-even-no-change-in-index-tp4074965.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1432811614768-4074965.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9kFwpxBaDZt0C_HzUG_U5zHpTWgGBqfReEzrB0WK1umg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9kFwpxBaDZt0C_HzUG_U5zHpTWgGBqfReEzrB0WK1umg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAECdJzBpOYsTwB9F6w6nNWRp_nCHmQfZy0fbeFy484Q9969Zog%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4