# Elasticsearch & mongo - sync **URL:** \ **Category:** Elasticsearch\ **Created:** [August 3, 2010, 7:01pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184 "2010-08-03T19:01:12Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![John\_Merrells](https://avatars.discourse-cdn.com/v4/letter/j/bcef8e/32.png) [@John\_Merrells](https://discuss.elastic.co/u/John_Merrells)\ **Post date:** [August 3, 2010, 7:01pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/1 "2010-08-03T19:01:12Z")
Hello, I have a bunch of documents stored in mongo that I'm indexing with elasticsearch. Would there happen to be a handy sync script/tool to keep elastic search up to date wrt a mongo container...? Anyone on the list been down this road before and care to share some wisdom? John ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 4, 2010, 5:55am UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/2 "2010-08-04T05:55:27Z")
I know that some expressed interest in it. I don't have much mongo experience (actually, none...), but some suggestions can be either hook into a "post commit" hook in mongo, or query it in some way for latest changes? Another option, which is more on the application side, is to apply the same changes done to mongo to elasticsearch as well. Sure you thought of these already ... -shay.banon On Tue, Aug 3, 2010 at 10:01 PM, John Merrells [merrells@gmail.com](mailto:merrells@gmail.com) wrote: > Hello, > > I have a bunch of documents stored in mongo that I'm indexing with > elasticsearch. > Would there happen to be a handy sync script/tool to keep Elasticsearch up > to > date wrt a mongo container...? Anyone on the list been down this road > before and > care to share some wisdom? > > John ---
**Author:** ![John\_Merrells](https://avatars.discourse-cdn.com/v4/letter/j/bcef8e/32.png) [@John\_Merrells](https://discuss.elastic.co/u/John_Merrells)\ **Post date:** [August 4, 2010, 1:26pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/3 "2010-08-04T13:26:47Z")
On Aug 3, 2010, at 10:55 PM, Shay Banon wrote: > I know that some expressed interest in it. I don't have much mongo experience (actually, none...), but some suggestions can be either hook into a "post commit" hook in mongo, or query it in some way for latest changes? Another option, which is more on the application side, is to apply the same changes done to mongo to elasticsearch as well. Sure you thought of these already ... My curent plan is to have an updated\_at datetime field on each document. With that I can ask Elasticsearch what the max datetime it has and then on mongo ask for all the documents with datetime greater than that. The only twisty bit so far is that the datetime is going to need to be stored as a string on mongo, and a long on elasticsearch, but in theory it should work. John ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 4, 2010, 1:31pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/4 "2010-08-04T13:31:37Z")
Sounds good. There is no long type in mongo? On Wed, Aug 4, 2010 at 4:26 PM, John Merrells [merrells@gmail.com](mailto:merrells@gmail.com) wrote: > On Aug 3, 2010, at 10:55 PM, Shay Banon wrote: > > > I know that some expressed interest in it. I don't have much mongo > > experience (actually, none...), but some suggestions can be either hook into > > a "post commit" hook in mongo, or query it in some way for latest changes? > > Another option, which is more on the application side, is to apply the same > > changes done to mongo to elasticsearch as well. Sure you thought of these > > already ... > > My curent plan is to have an updated\_at datetime field on each document. > With that I can ask Elasticsearch what the max datetime it has and then on > mongo ask for all the documents with datetime greater than that. The only > twisty bit so far is that the datetime is going to need to be stored as a > string > on mongo, and a long on elasticsearch, but in theory it should work. > > John ---
**Author:** ![John\_Merrells](https://avatars.discourse-cdn.com/v4/letter/j/bcef8e/32.png) [@John\_Merrells](https://discuss.elastic.co/u/John_Merrells)\ **Post date:** [August 4, 2010, 1:38pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/5 "2010-08-04T13:38:33Z")
On Aug 4, 2010, at 6:31 AM, Shay Banon wrote: > Sounds good. There is no long type in mongo? Doh. Yes, it does. That'd be simpler. I got there because I started looking at the Date type, but on mongo its hard to query it through the protocol, and on elasticsearch there's no max for date... so then I tried string.... then long.... so yeah I should just use long on both.... John ---
**Author:** ![John\_Merrells](https://avatars.discourse-cdn.com/v4/letter/j/bcef8e/32.png) [@John\_Merrells](https://discuss.elastic.co/u/John_Merrells)\ **Post date:** [August 4, 2010, 1:48pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/6 "2010-08-04T13:48:42Z")
On Aug 4, 2010, at 6:31 AM, Shay Banon wrote: > Sounds good. There is no long type in mongo? Sorry to bore everyone with my early morning pre-coffee mumblings, but 'float' on both mongo and elastic is most probably the way to go. John ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 4, 2010, 1:49pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/7 "2010-08-04T13:49:51Z")
Why float? I assumed long and represent it as a timestamp. Date type in elasticsearch is just a facade on top of long. -shay.bano On Wed, Aug 4, 2010 at 4:48 PM, John Merrells [merrells@gmail.com](mailto:merrells@gmail.com) wrote: > On Aug 4, 2010, at 6:31 AM, Shay Banon wrote: > > > Sounds good. There is no long type in mongo? > > Sorry to bore everyone with my early morning pre-coffee mumblings, > but 'float' on both mongo and elastic is most probably the way to go. > > John ---
**Author:** ![John\_Merrells](https://avatars.discourse-cdn.com/v4/letter/j/bcef8e/32.png) [@John\_Merrells](https://discuss.elastic.co/u/John_Merrells)\ **Post date:** [August 4, 2010, 1:59pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/8 "2010-08-04T13:59:42Z")
On Aug 4, 2010, at 6:49 AM, Shay Banon wrote: > Why float? I assumed long and represent it as a timestamp. Date type in elasticsearch is just a facade on top of long. There could be many updates in a second... In Ruby Time.now.to\_f returns a Float.... I've just realized that mongo can auto generate ids, which are a bit like guids, so have a seconds and a counter field within them.... soo... I could extract those bits and use them, but it'd amount to much the same thing, and be a bit opaque. Still pre coffee.... still rambling.... John ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 4, 2010, 2:03pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/9 "2010-08-04T14:03:39Z")
Auto increment GUIDs are the best, but with timestamps, as you suggested, there might be severa within the same resolution. One way to work around them is the create the query with where you subtract the resolution you get on your machine (1 milli for example), use "index" on whatever falls within it, and use create on the rest (assuming you know that they don't exists in elasticsearch). By the way, the most difficult part when it comes to sync's is to handle deletes, I assume you don't have them? -shay.banon On Wed, Aug 4, 2010 at 4:59 PM, John Merrells [merrells@gmail.com](mailto:merrells@gmail.com) wrote: > On Aug 4, 2010, at 6:49 AM, Shay Banon wrote: > > > Why float? I assumed long and represent it as a timestamp. Date type in > > elasticsearch is just a facade on top of long. > > There could be many updates in a second... In Ruby Time.now.to\_f returns a > Float.... > > I've just realized that mongo can auto generate ids, which are a bit like > guids, so have a > seconds and a counter field within them.... soo... I could extract those > bits and use them, > but it'd amount to much the same thing, and be a bit opaque. > > Still pre coffee.... still rambling.... > > John ---
**Author:** ![John\_Merrells](https://avatars.discourse-cdn.com/v4/letter/j/bcef8e/32.png) [@John\_Merrells](https://discuss.elastic.co/u/John_Merrells)\ **Post date:** [August 4, 2010, 2:30pm UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/10 "2010-08-04T14:30:40Z")
On Aug 4, 2010, at 7:03 AM, Shay Banon wrote: > By the way, the most difficult part when it comes to sync's is to handle deletes, I assume you don't have them? No deletes. John ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:21am UTC](https://discuss.elastic.co/t/elasticsearch-mongo-sync/3184/11 "2017-07-06T04:21:11Z")