# Batch doc updates and real-time search **URL:** \ **Category:** Elasticsearch\ **Created:** [June 10, 2010, 7:55pm UTC](https://discuss.elastic.co/t/batch-doc-updates-and-real-time-search/3010 "2010-06-10T19:55:05Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 10, 2010, 7:55pm UTC](https://discuss.elastic.co/t/batch-doc-updates-and-real-time-search/3010/1 "2010-06-10T19:55:05Z")
Hello, Does ES have anything in it that makes it better than Lucene when one needs to modify a large number of docs (e.g. modify a "tags" field for 50K documents in the result set) and see the changes reflected in real- time? With straight Lucene, one could get the NRT part, but modifying 50K docs would trigger 50K doc deletes and 50K doc adds for just slightly modified documents. Does ES happen to have a solution for this? ## Thanks, Otis Sematext -- [http://sematext.com/](http://sematext.com/) -- Solr - Lucene - Nutch ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 10, 2010, 8:03pm UTC](https://discuss.elastic.co/t/batch-doc-updates-and-real-time-search/3010/2 "2010-06-10T20:03:27Z")
Hi, No, ES does not handle it and you will need to do a full update to change a certain field (like rename a tag). There are hacks to do it on top of Lucene (for example, by maintaining two parallel indices), but they are not really manageable. Not really sure how NRT fits into this? Because of the deletes and the cloning? If that is the case, then NRT is only opened in a scheduled manner (though there is an API for that). The nice thing is that this will be much much faster since you go distributed and you basically spread the load. -shay.banon On Thu, Jun 10, 2010 at 10:55 PM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Hello, > > Does ES have anything in it that makes it better than Lucene when one > needs to modify a large number of docs (e.g. modify a "tags" field for > 50K documents in the result set) and see the changes reflected in real- > time? > > With straight Lucene, one could get the NRT part, but modifying 50K > docs would trigger 50K doc deletes and 50K doc adds for just slightly > modified documents. > > Does ES happen to have a solution for this? > > ## Thanks, Otis > > Sematext -- [http://sematext.com/](http://sematext.com/) -- Solr - Lucene - Nutch ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 10, 2010, 8:24pm UTC](https://discuss.elastic.co/t/batch-doc-updates-and-real-time-search/3010/3 "2010-06-10T20:24:45Z")
Thanks Shay, More below. On Jun 10, 4:03 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi, > > No, ES does not handle it and you will need to do a full update to change a > certain field (like rename a tag). There are hacks to do it on top of Lucene > (for example, by maintaining two parallel indices), but they are not really > manageable. Not really sure how NRT fits into this? Because of the deletes > and the cloning? If that is the case, then NRT is only opened in a scheduled > manner (though there is an API for that). > > The nice thing is that this will be much much faster since you go > distributed and you basically spread the load. Could you please expand on this a bit? Since I didn't mention distributed search, I wonder what you are referring to. Are you saying that IF I were to involve multiple shards (and thus multiple nodes/servers), then batch doc updates would be faster because, since docs would be spread over multiple nodes, the overall time needed to update a large batch of docs would be shorter because updates of sub-sets of docs would happen in parallel on multiple nodes? ## Thanks, Otis Sematext --[http://sematext.com/--](http://sematext.com/--) Solr - Lucene - Nutch > On Thu, Jun 10, 2010 at 10:55 PM, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote: > > > Hello, > > > Does ES have anything in it that makes it better than Lucene when one > > needs to modify a large number of docs (e.g. modify a "tags" field for > > 50K documents in the result set) and see the changes reflected in real- > > time? > > > With straight Lucene, one could get the NRT part, but modifying 50K > > docs would trigger 50K doc deletes and 50K doc adds for just slightly > > modified documents. > > > Does ES happen to have a solution for this? > > > ## Thanks, Otis > > > > Sematext --[http://sematext.com/--](http://sematext.com/--) Solr - Lucene - Nutch ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 10, 2010, 8:34pm UTC](https://discuss.elastic.co/t/batch-doc-updates-and-real-time-search/3010/4 "2010-06-10T20:34:19Z")
Yep, I meant that the indexing process or update process would be spread across several nodes, thus will be faster. Sadly, there is no simple solution for this in the Lucene world as far as I know... . -shay.banon On Thu, Jun 10, 2010 at 11:24 PM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Thanks Shay, > > More below. > > On Jun 10, 4:03 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hi, > > > > No, ES does not handle it and you will need to do a full update to > > change a > > certain field (like rename a tag). There are hacks to do it on top of > > Lucene > > (for example, by maintaining two parallel indices), but they are not > > really > > manageable. Not really sure how NRT fits into this? Because of the > > deletes > > and the cloning? If that is the case, then NRT is only opened in a > > scheduled > > manner (though there is an API for that). > > > > The nice thing is that this will be much much faster since you go > > distributed and you basically spread the load. > > Could you please expand on this a bit? Since I didn't mention > distributed search, I wonder what you are referring to. > > Are you saying that IF I were to involve multiple shards (and thus > multiple nodes/servers), then batch doc updates would be faster > because, since docs would be spread over multiple nodes, the overall > time needed to update a large batch of docs would be shorter because > updates of sub-sets of docs would happen in parallel on multiple > nodes? > > ## Thanks, Otis > > Sematext --[http://sematext.com/--](http://sematext.com/--) Solr - Lucene - Nutch > > > On Thu, Jun 10, 2010 at 10:55 PM, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) > > wrote: > > > > > Hello, > > > > > Does ES have anything in it that makes it better than Lucene when one > > > needs to modify a large number of docs (e.g. modify a "tags" field for > > > 50K documents in the result set) and see the changes reflected in real- > > > time? > > > > > With straight Lucene, one could get the NRT part, but modifying 50K > > > docs would trigger 50K doc deletes and 50K doc adds for just slightly > > > modified documents. > > > > > Does ES happen to have a solution for this? > > > > > ## Thanks, Otis > > > > > > Sematext --[http://sematext.com/--](http://sematext.com/--) Solr - Lucene - Nutch ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:23am UTC](https://discuss.elastic.co/t/batch-doc-updates-and-real-time-search/3010/5 "2017-07-06T04:23:24Z")