# Incremental Indexing Stragegy **URL:** \ **Category:** Elasticsearch\ **Created:** [March 8, 2012, 2:34am UTC](https://discuss.elastic.co/t/incremental-indexing-stragegy/6940 "2012-03-08T02:34:49Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Aubrey\_Trey\_Rhodes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aubrey_trey_rhodes/32/2966_2.png) [@Aubrey\_Trey\_Rhodes](https://discuss.elastic.co/u/Aubrey_Trey_Rhodes)\ **Post date:** [March 8, 2012, 2:34am UTC](https://discuss.elastic.co/t/incremental-indexing-stragegy/6940/1 "2012-03-08T02:34:49Z")
Hi All, I'm working on an application that will index the results of api calls from a few different services. Right now I'm not indexing that much data (less than 1000 documents), so when I want to include new information in the index, I just throw out the whole thing and rebuild it from scratch. I want to begin to scale up the application and because of limitations with the api's, it will no longer be possible to make the calls for all the documents each time I want to update the index. I'm trying to figure out a strategy for incremental updating using elastic search so that I only grab new information that is not indexed. Each of the results have an associated timestamp, so I was going to use that to base the updating off of, but I'm wondering how to get the maximum timestamp for the documents in my index. It looks like I could add a field to the documents that is an integer that represents the timestamp. From that I could run a query using the timestamp integer as the custom score and limit the results to 1 to get the maximum. Is that the best way to get the maximum value of an integer field with elasticsearch? How do most applications handle this? Do they just make use of an external datastore for these types of values? Thanks! Aubrey Rhodes ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [March 8, 2012, 6:49am UTC](https://discuss.elastic.co/t/incremental-indexing-stragegy/6940/2 "2012-03-08T06:49:08Z")
Hi Aubrey, I'm not sure if I understood your scenario well enough, and I'm not exactly an Elasticsearch expert, but here's some information that might help: - you can enable Elasticsearch to automatically add a timestamp for your documents for the time they were inserted. You can also add that field in your document and point \_timestamp to it. More details here: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/timestamp-field.html) This allows you to do all sorts of cool stuff, like: - provided that you could get some sort of version for your api calls, you could use versioning in Elasticsearch: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2011/02/08/versioning.html) On Mar 8, 4:34 am, "Aubrey "Trey" Rhodes" [aubrey.c.rho...@gmail.com](mailto:aubrey.c.rho...@gmail.com) wrote: > Hi All, > > I'm working on an application that will index the results of api calls from > a few different services. Right now I'm not indexing that much data (less > than 1000 documents), so when I want to include new information in the > index, I just throw out the whole thing and rebuild it from scratch. I want > to begin to scale up the application and because of limitations with the > api's, it will no longer be possible to make the calls for all the > documents each time I want to update the index. > > I'm trying to figure out a strategy for incremental updating using elastic > search so that I only grab new information that is not indexed. Each of the > results have an associated timestamp, so I was going to use that to base > the updating off of, but I'm wondering how to get the maximum timestamp for > the documents in my index. > > It looks like I could add a field to the documents that is an integer that > represents the timestamp. From that I could run a query using the timestamp > integer as the custom score and limit the results to 1 to get the maximum. > Is that the best way to get the maximum value of an integer field with > elasticsearch? > > How do most applications handle this? Do they just make use of an external > datastore for these types of values? > > Thanks! > Aubrey Rhodes ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 9, 2012, 6:42pm UTC](https://discuss.elastic.co/t/incremental-indexing-stragegy/6940/3 "2012-03-09T18:42:31Z")
I don't understand the possible problem as well…, why not just index the new docs? On Thursday, March 8, 2012 at 8:49 AM, Radu Gheorghe wrote: > Hi Aubrey, > > I'm not sure if I understood your scenario well enough, and I'm not > exactly an Elasticsearch expert, but here's some information that > might help: > > - you can enable Elasticsearch to automatically add a timestamp for > your documents for the time they were inserted. You can also add that > field in your document and point \_timestamp to it. More details here: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/timestamp-field.html) > This allows you to do all sorts of cool stuff, like: > > - sort documents on that field. To get the maximum value, you can do > a "match\_all" query, then sort by timestamp and get the number of > results to 1, as you said > > - do range queries (eg: I want all documents inserted in the last > hour) > > - put a TTL on documents (eg: only keep documents from the last two > days). More details on TTL and other stuff here: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/index_.html) > > - provided that you could get some sort of version for your api calls, > you could use versioning in Elasticsearch: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2011/02/08/versioning.html) > > On Mar 8, 4:34 am, "Aubrey "Trey" Rhodes" > \<[aubrey.c.rho...@gmail.com](mailto:aubrey.c.rho...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote: > > > Hi All, > > > > I'm working on an application that will index the results of api calls from > > a few different services. Right now I'm not indexing that much data (less > > than 1000 documents), so when I want to include new information in the > > index, I just throw out the whole thing and rebuild it from scratch. I want > > to begin to scale up the application and because of limitations with the > > api's, it will no longer be possible to make the calls for all the > > documents each time I want to update the index. > > > > I'm trying to figure out a strategy for incremental updating using elastic > > search so that I only grab new information that is not indexed. Each of the > > results have an associated timestamp, so I was going to use that to base > > the updating off of, but I'm wondering how to get the maximum timestamp for > > the documents in my index. > > > > It looks like I could add a field to the documents that is an integer that > > represents the timestamp. From that I could run a query using the timestamp > > integer as the custom score and limit the results to 1 to get the maximum. > > Is that the best way to get the maximum value of an integer field with > > elasticsearch? > > > > How do most applications handle this? Do they just make use of an external > > datastore for these types of values? > > > > Thanks! > > Aubrey Rhodes ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:36am UTC](https://discuss.elastic.co/t/incremental-indexing-stragegy/6940/4 "2017-07-06T03:36:35Z")