# Frequently updated int field **URL:** \ **Category:** Elasticsearch\ **Created:** [June 7, 2011, 3:12am UTC](https://discuss.elastic.co/t/frequently-updated-int-field/4562 "2011-06-07T03:12:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Andy\_2](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Andy\_2](https://discuss.elastic.co/u/Andy_2)\ **Post date:** [June 7, 2011, 3:12am UTC](https://discuss.elastic.co/t/frequently-updated-int-field/4562/1 "2011-06-07T03:12:20Z")
I have an integer field "popularity" that is frequently updated. The "popularity" of a document can either be increased or decreased. I use the value of that field to help rank my search results. A search engine might not be designed for frequently updated fields like that. Any tips on how best to handle that in ElasticSearch? Thanks. ---
**Author:** ![Mahendra\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahendra_m/32/3128_2.png) [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)\ **Post date:** [June 7, 2011, 4:42am UTC](https://discuss.elastic.co/t/frequently-updated-int-field/4562/2 "2011-06-07T04:42:02Z")
Hi Andy, On Tue, Jun 7, 2011 at 8:42 AM, Andy [selforganized@gmail.com](mailto:selforganized@gmail.com) wrote: > I have an integer field "popularity" that is frequently updated. The > "popularity" of a document can either be increased or decreased. I use > the value of that field to help rank my search results. Even I have the same use case. A "popularity" field being updated based on usage of a document. > A search engine might not be designed for frequently updated fields > like that. Any tips on how best to handle that in Elasticsearch? How frequent are your updates ? Elasticsearch, I think, can handle frequent updates pretty well. Do have a look at this link - [http://engineering.socialcast.com/2011/05/realtime-search-solr-vs-elasticsearch/](http://engineering.socialcast.com/2011/05/realtime-search-solr-vs-elasticsearch/) Even so, we use some other tricks to reduce the frequency of updates. Instead of updating Elasticsearch frequently, we collect the usage of a document over a period of time (say 10 minutes) aggregate the result and then update it to Elasticsearch. Maybe you can look at a similar approach. Regards, Mahendra [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![Andy\_2](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Andy\_2](https://discuss.elastic.co/u/Andy_2)\ **Post date:** [June 7, 2011, 5:56pm UTC](https://discuss.elastic.co/t/frequently-updated-int-field/4562/3 "2011-06-07T17:56:11Z")
Thanks Mahendra. How do you implement your batched updates? Do you fire up a cron job every x minutes to get the "popularity" values from a database and then use it to update Elasticsearch? On Jun 7, 12:42 am, Mahendra M [mahendr...@gmail.com](mailto:mahendr...@gmail.com) wrote: > Hi Andy, > > On Tue, Jun 7, 2011 at 8:42 AM, Andy [selforgani...@gmail.com](mailto:selforgani...@gmail.com) wrote: > > > I have an integer field "popularity" that is frequently updated. The > > "popularity" of a document can either be increased or decreased. I use > > the value of that field to help rank my search results. > > Even I have the same use case. A "popularity" field being updated > based on usage of a document. > > > A search engine might not be designed for frequently updated fields > > like that. Any tips on how best to handle that in Elasticsearch? > > How frequent are your updates ? Elasticsearch, I think, can handle > frequent updates pretty well. > Do have a look at this link -[http://engineering.socialcast.com/2011/05/realtime-search-solr-vs-ela](http://engineering.socialcast.com/2011/05/realtime-search-solr-vs-ela)... > > Even so, we use some other tricks to reduce the frequency of updates. > Instead of updating Elasticsearch frequently, we collect the usage of > a document over a period of time (say 10 minutes) aggregate the result > and then update it to Elasticsearch. Maybe you can look at a similar > approach. > > Regards, > Mahendra > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [June 7, 2011, 7:19pm UTC](https://discuss.elastic.co/t/frequently-updated-int-field/4562/4 "2011-06-07T19:19:31Z")
Hey Andy/Mahendra, We do the exact same things that Mahendra mentions. We have a custom data processing tool we use instead of cron, but it runs in a very similar fashion. It works off of relative values, though, where we get the number of document requests since the last run and only update documents that have changed. On top of doing these updates in batch at certain intervals, we are also considering ignoring documents with only a request or two. We had hoped that parent/child documents would allow us to do this more efficiently, but parent documents cannot be sorted by values in child documents. We haven't yet launched anything using this, but don't expect any issues. I had seen some interesting discussions around this at the Lucene level, but don't believe any of it pertains to ES: [http://www.lucenerevolution.org/blog/2011/05/31/224/](http://www.lucenerevolution.org/blog/2011/05/31/224/) [http://www.mjohnston.com/2009/09/adding-external-datasources-to-lucene-scoring/](http://www.mjohnston.com/2009/09/adding-external-datasources-to-lucene-scoring/) (A little older, so not sure if still relvant) Thanks, Paul On Jun 7, 11:56 am, Andy [selforgani...@gmail.com](mailto:selforgani...@gmail.com) wrote: > Thanks Mahendra. > > How do you implement your batched updates? Do you fire up a cron job > every x minutes to get the "popularity" values from a database and > then use it to update Elasticsearch? > > On Jun 7, 12:42 am, Mahendra M [mahendr...@gmail.com](mailto:mahendr...@gmail.com) wrote: > > > Hi Andy, > > > On Tue, Jun 7, 2011 at 8:42 AM, Andy [selforgani...@gmail.com](mailto:selforgani...@gmail.com) wrote: > > > > > I have an integer field "popularity" that is frequently updated. The > > > "popularity" of a document can either be increased or decreased. I use > > > the value of that field to help rank my search results. > > > Even I have the same use case. A "popularity" field being updated > > based on usage of a document. > > > > A search engine might not be designed for frequently updated fields > > > like that. Any tips on how best to handle that in Elasticsearch? > > > How frequent are your updates ? Elasticsearch, I think, can handle > > frequent updates pretty well. > > Do have a look at this link -[http://engineering.socialcast.com/2011/05/realtime-search-solr-vs-ela](http://engineering.socialcast.com/2011/05/realtime-search-solr-vs-ela)... > > > Even so, we use some other tricks to reduce the frequency of updates. > > Instead of updating Elasticsearch frequently, we collect the usage of > > a document over a period of time (say 10 minutes) aggregate the result > > and then update it to Elasticsearch. Maybe you can look at a similar > > approach. > > > Regards, > > Mahendra > > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![fashionalwallet](https://avatars.discourse-cdn.com/v4/letter/f/839c29/32.png) [@fashionalwallet](https://discuss.elastic.co/u/fashionalwallet)\ **Post date:** [June 10, 2011, 12:30am UTC](https://discuss.elastic.co/t/frequently-updated-int-field/4562/5 "2011-06-10T00:30:38Z")
- deleted - ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:04am UTC](https://discuss.elastic.co/t/frequently-updated-int-field/4562/6 "2017-07-06T04:04:06Z")