# Persistent transaction log **URL:** \ **Category:** Elasticsearch\ **Created:** [August 9, 2010, 4:27am UTC](https://discuss.elastic.co/t/persistent-transaction-log/3198 "2010-08-09T04:27:08Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![talsalmona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talsalmona/32/3316_2.png) [@talsalmona](https://discuss.elastic.co/u/talsalmona)\ **Post date:** [August 9, 2010, 4:27am UTC](https://discuss.elastic.co/t/persistent-transaction-log/3198/1 "2010-08-09T04:27:08Z")
Hi, Is there a way to somehow persist the transaction log to be "replayed" at a later times? I think that in many situations when re-indexing is required (as a result of some catastrophe) it would be nice to be able to replay the transaction log (maybe from the last X hours). A more specific example: I use elastic search to index some content in the database. I have triggers that fire an index request when something is updated in the database (it's important for me to have the data in ES reflect the database as closely as possible). To support disaster recovery I backup the database and snapshot the ES gateway files every once in a while. When I want to recover, I restore the database and the gateway. The problem is that the database backup and gateway snapshot have not been created at the exact same time so there may be some missing data in ES. What I do to solve this is to re-index (from the database) the last X hours to make sure ES and the database are in sync. After this long story... I was thinking that it would be nicer if I could somehow replay a persistent transaction log instead of hitting the database to get the data. Is this somehow possible? Thanks, Tal ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 9, 2010, 2:17pm UTC](https://discuss.elastic.co/t/persistent-transaction-log/3198/2 "2010-08-09T14:17:40Z")
Hi, Elasticsearch does not store a boundless transaction log, it only stores a transaction log till the next flush is executed (changes are "committed" to the index). It also, by default, does a flush every 5000 operations (per shard). In general, you should let the snapshotting take place every short interval of time (by default its 10 seconds). The snapshotting is delta snapshots, so in the general case, there won't be a lot of data to sync. -shay.banon On Mon, Aug 9, 2010 at 7:27 AM, Tal [talsalmona@gmail.com](mailto:talsalmona@gmail.com) wrote: > Hi, > > Is there a way to somehow persist the transaction log to be "replayed" > at a later times? > I think that in many situations when re-indexing is required (as a > result of some catastrophe) it would be nice to be able to replay the > transaction log (maybe from the last X hours). > > A more specific example: I use Elasticsearch to index some content in > the database. I have triggers that fire an index request when > something is updated in the database (it's important for me to have > the data in ES reflect the database as closely as possible). > To support disaster recovery I backup the database and snapshot the ES > gateway files every once in a while. > When I want to recover, I restore the database and the gateway. > The problem is that the database backup and gateway snapshot have not > been created at the exact same time so there may be some missing data > in ES. > What I do to solve this is to re-index (from the database) the last X > hours to make sure ES and the database are in sync. > After this long story... I was thinking that it would be nicer if I > could somehow replay a persistent transaction log instead of hitting > the database to get the data. > Is this somehow possible? > > Thanks, > Tal ---
**Author:** ![talsalmona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talsalmona/32/3316_2.png) [@talsalmona](https://discuss.elastic.co/u/talsalmona)\ **Post date:** [August 10, 2010, 5:09am UTC](https://discuss.elastic.co/t/persistent-transaction-log/3198/3 "2010-08-10T05:09:13Z")
I see. OK, thanks. Tal On Aug 9, 5:17 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi, > > Elasticsearch does not store a boundless transaction log, it only stores a > transaction log till the next flush is executed (changes are "committed" to > the index). It also, by default, does a flush every 5000 operations (per > shard). > > In general, you should let the snapshotting take place every short > interval of time (by default its 10 seconds). The snapshotting is delta > snapshots, so in the general case, there won't be a lot of data to sync. > > -shay.banon > > On Mon, Aug 9, 2010 at 7:27 AM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote: > > > Hi, > > > Is there a way to somehow persist the transaction log to be "replayed" > > at a later times? > > I think that in many situations when re-indexing is required (as a > > result of some catastrophe) it would be nice to be able to replay the > > transaction log (maybe from the last X hours). > > > A more specific example: I use Elasticsearch to index some content in > > the database. I have triggers that fire an index request when > > something is updated in the database (it's important for me to have > > the data in ES reflect the database as closely as possible). > > To support disaster recovery I backup the database and snapshot the ES > > gateway files every once in a while. > > When I want to recover, I restore the database and the gateway. > > The problem is that the database backup and gateway snapshot have not > > been created at the exact same time so there may be some missing data > > in ES. > > What I do to solve this is to re-index (from the database) the last X > > hours to make sure ES and the database are in sync. > > After this long story... I was thinking that it would be nicer if I > > could somehow replay a persistent transaction log instead of hitting > > the database to get the data. > > Is this somehow possible? > > > Thanks, > > Tal ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:20am UTC](https://discuss.elastic.co/t/persistent-transaction-log/3198/4 "2017-07-06T04:20:59Z")