# River to elasticSearch based on TTL **URL:** \ **Category:** Elasticsearch\ **Created:** [October 27, 2011, 2:23pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707 "2011-10-27T14:23:58Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 27, 2011, 2:23pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/1 "2011-10-27T14:23:58Z")
Hi , If i got it right , with the new TTL feature , a timeStamp of creation would be tagged with each document. If this is right cant we create a river to elasticSearch based on this. That is each time we poll elasticSearch for changes , we will ask it like "provide me all changes since this timeStamp" . There might be some overlapping (very few) documents for that milliSecond but then we can discard it at the river code. Thanks Vineeth ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 27, 2011, 2:28pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/2 "2011-10-27T14:28:24Z")
If you are right, it's a very good idea ! David 😉 Le 27 oct. 2011 à 16:23, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) a écrit : > Hi , > > If i got it right , with the new TTL feature , a timeStamp of creation would be tagged with each document. > If this is right cant we create a river to elasticSearch based on this. > > That is each time we poll elasticSearch for changes , we will ask it like "provide me all changes since this timeStamp" . > There might be some overlapping (very few) documents for that milliSecond but then we can discard it at the river code. > > Thanks > Vineeth ---
**Author:** ![Benjamin\_Deveze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamin_deveze/32/1577_2.png) [@Benjamin\_Deveze](https://discuss.elastic.co/u/Benjamin_Deveze)\ **Post date:** [October 27, 2011, 2:33pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/3 "2011-10-27T14:33:32Z")
Hi, if you don't provide a timestamp when indexing/updating a document and you enable timestamp in the mapping then each document will be associated with a timestamp corresponding to indexing time. The timestamp field is indexed by default and can be queried as a date field if I remember well. So if you have a synchronized time on your cluster you are able to query all the docs that have changed since a specified moment. So I guess it could work. ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 27, 2011, 2:46pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/4 "2011-10-27T14:46:19Z")
We dont have to synchronize the time. The endpoint (in this case ES) will have to remember which timeStamp did it fetch it last time. Then do a "get me all since this timestamp" query. Thanks Vineeth On Thu, Oct 27, 2011 at 8:03 PM, Benjamin Devèze [benjamin.deveze@gmail.com](mailto:benjamin.deveze@gmail.com)wrote: > Hi, > > if you don't provide a timestamp when indexing/updating a document and you > enable timestamp in the mapping then each document will be associated with a > timestamp corresponding to indexing time. > The timestamp field is indexed by default and can be queried as a date > field if I remember well. So if you have a synchronized time on your cluster > you are able to query all the docs that have > changed since a specified moment. > > So I guess it could work. ---
**Author:** ![Benjamin\_Deveze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamin_deveze/32/1577_2.png) [@Benjamin\_Deveze](https://discuss.elastic.co/u/Benjamin_Deveze)\ **Post date:** [October 27, 2011, 3:28pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/5 "2011-10-27T15:28:28Z")
If mean if you want a "get me all since this timestamp" query on a distributed environment and you rely on automatic timestamp of ES you have to make sure that all your nodes have the same time. ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 27, 2011, 3:32pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/6 "2011-10-27T15:32:11Z")
oh ok.... yes that makes sense. But if some work around is not there in the current implementation of deletion , how are things like "delete a document based on TTL even if a node was taken out the cluster and a new one is put back and the new one is not time synced up with first one" handleled ? Thanks Vineeth On Thu, Oct 27, 2011 at 8:58 PM, Benjamin Devèze [benjamin.deveze@gmail.com](mailto:benjamin.deveze@gmail.com)wrote: > If mean if you want a "get me all since this timestamp" query on a > distributed environment and you rely on automatic timestamp of ES you have > to make sure that all your nodes have the same time. ---
**Author:** ![Benjamin\_Deveze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamin_deveze/32/1577_2.png) [@Benjamin\_Deveze](https://discuss.elastic.co/u/Benjamin_Deveze)\ **Post date:** [October 27, 2011, 3:50pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/7 "2011-10-27T15:50:04Z")
The TTL feature assume that the time is synced up cluster wide ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 27, 2011, 4:23pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/8 "2011-10-27T16:23:07Z")
that means that if the time is not synced up and if some reordering happens we might loose a document before it "actually" expires. Looks like a bug to me. Thanks Vineeth On Thu, Oct 27, 2011 at 9:20 PM, Benjamin Devèze [benjamin.deveze@gmail.com](mailto:benjamin.deveze@gmail.com)wrote: > The TTL feature assume that the time is synced up cluster wide ---
**Author:** ![Electric\_Owl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/electric_owl/32/3094_2.png) [@Electric\_Owl](https://discuss.elastic.co/u/Electric_Owl)\ **Post date:** [October 27, 2011, 4:38pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/9 "2011-10-27T16:38:49Z")
On Oct 27, 5:23 pm, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > that means that if the time is not synced up and if some reordering happens > we might loose a document before it "actually" expires. > Looks like a bug to me. Presumably if you are operating a cluster you will have all the servers getting their time from the same NTP server and syncing on a regular basis. I couldn't say whether it is a bug or not but maybe you need to consider your system/network architecture first before declaring it so. Ian > Thanks > Vineeth > > On Thu, Oct 27, 2011 at 9:20 PM, Benjamin Devèze > [benjamin.dev...@gmail.com](mailto:benjamin.dev...@gmail.com)wrote: > > > The TTL feature assume that the time is synced up cluster wide ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 27, 2011, 4:48pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/10 "2011-10-27T16:48:27Z")
I have operates such large scale systems which are backed by NTP. I have also seen NTP deomen going down without coming back too. Also it was kinda common that some of the newly added machines didn't have NTP running on it. (yes , even after using configuration engines like cfEngine , etch etc) As the risk here is deletion rather than some other trivial issues , i had this concern. Thanks Vineeth On Thu, Oct 27, 2011 at 10:08 PM, Electric Owl [ian.lewis.65@gmail.com](mailto:ian.lewis.65@gmail.com)wrote: > On Oct 27, 5:23 pm, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > > > that means that if the time is not synced up and if some reordering > > happens > > we might loose a document before it "actually" expires. > > Looks like a bug to me. > > Presumably if you are operating a cluster you will have all the > servers getting their time from the same NTP server and syncing on a > regular basis. > > I couldn't say whether it is a bug or not but maybe you need to > consider your system/network architecture first before declaring it > so. > > Ian > > > Thanks > > Vineeth > > > > On Thu, Oct 27, 2011 at 9:20 PM, Benjamin Devèze > > [benjamin.dev...@gmail.com](mailto:benjamin.dev...@gmail.com)wrote: > > > > > The TTL feature assume that the time is synced up cluster wide ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 27, 2011, 4:53pm UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/11 "2011-10-27T16:53:43Z")
To add to this , i have also seen time moving at a faster rates in a virtual machine (yes it was a bug in that virtual machine creation application) Thanks Vineeth On Thu, Oct 27, 2011 at 10:18 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > I have operates such large scale systems which are backed by NTP. > I have also seen NTP deomen going down without coming back too. > Also it was kinda common that some of the newly added machines didn't have > NTP running on it. (yes , even after using > configuration engines like cfEngine , etch etc) > As the risk here is deletion rather than some other trivial issues , i had > this concern. > > Thanks > Vineeth > > On Thu, Oct 27, 2011 at 10:08 PM, Electric Owl [ian.lewis.65@gmail.com](mailto:ian.lewis.65@gmail.com)wrote: > > > On Oct 27, 5:23 pm, Vineeth Mohan [vineethmo...@algotree.com](mailto:vineethmo...@algotree.com) wrote: > > > > > that means that if the time is not synced up and if some reordering > > > happens > > > we might loose a document before it "actually" expires. > > > Looks like a bug to me. > > > > Presumably if you are operating a cluster you will have all the > > servers getting their time from the same NTP server and syncing on a > > regular basis. > > > > I couldn't say whether it is a bug or not but maybe you need to > > consider your system/network architecture first before declaring it > > so. > > > > Ian > > > > > Thanks > > > Vineeth > > > > > > On Thu, Oct 27, 2011 at 9:20 PM, Benjamin Devèze > > > [benjamin.dev...@gmail.com](mailto:benjamin.dev...@gmail.com)wrote: > > > > > > > The TTL feature assume that the time is synced up cluster wide ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:50am UTC](https://discuss.elastic.co/t/river-to-elasticsearch-based-on-ttl/5707/12 "2017-07-06T03:50:38Z")