# Document pre-processor **URL:** \ **Category:** Elasticsearch\ **Created:** [July 19, 2010, 8:59pm UTC](https://discuss.elastic.co/t/document-pre-processor/3119 "2010-07-19T20:59:35Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [July 19, 2010, 8:59pm UTC](https://discuss.elastic.co/t/document-pre-processor/3119/1 "2010-07-19T20:59:35Z")
Hello, What is the best way to inject a custom document "processor" where one could manipulate the document and its fields before the document gets indexed? This is what Solr has, for example: [http://search-lucene.com/jd/solr/org/apache/solr/update/processor/UpdateRequestProcessor.html](http://search-lucene.com/jd/solr/org/apache/solr/update/processor/UpdateRequestProcessor.html) [http://wiki.apache.org/solr/UpdateRequestProcessor](http://wiki.apache.org/solr/UpdateRequestProcessor) ## Thanks, Otis Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch Lucene ecosystem search :: [http://search-lucene.com/](http://search-lucene.com/) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 19, 2010, 9:48pm UTC](https://discuss.elastic.co/t/document-pre-processor/3119/2 "2010-07-19T21:48:50Z")
There isn't really a hook point for this (i.e. get the content (json / xson) and munge it). It can be added, but I think that it make more sense to do it on the "client" side if one wish to do it, no? The fact that elasticsearch supports json like structures natively (with deep level objects) means that you can basically create that pre processor on your client side. Another option is to create your own mapper that handles the part of the data you want, but that also is I think more complicated than simply doing it on the client side. -shay.banon On Mon, Jul 19, 2010 at 11:59 PM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Hello, > > What is the best way to inject a custom document "processor" where one > could manipulate the document and its fields before the document gets > indexed? > > This is what Solr has, for example: > > [http://search-lucene.com/jd/solr/org/apache/solr/update/processor/UpdateRequestProcessor.html](http://search-lucene.com/jd/solr/org/apache/solr/update/processor/UpdateRequestProcessor.html) > [UpdateRequestProcessor - Solr - Apache Software Foundation](http://wiki.apache.org/solr/UpdateRequestProcessor) > > ## Thanks, Otis > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch > Lucene ecosystem search :: [http://search-lucene.com/](http://search-lucene.com/) ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [July 21, 2010, 3:52pm UTC](https://discuss.elastic.co/t/document-pre-processor/3119/3 "2010-07-21T15:52:53Z")
Hello, I looked at [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/index/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/index/) and at [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/engine/robin/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/engine/robin/) , but couldn't find anything that sounds like Solr's UpdateRequestProcessor. Am I not looking in the right places? Is this feature missing but already on the roadmap? Thanks, Otis On Jul 19, 4:59 pm, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote: > Hello, > > What is the best way to inject a custom document "processor" where one > could manipulate the document and its fields before the document gets > indexed? > > This is what Solr has, for example: > > [http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd...http://wiki.apache.org/solr/UpdateRequestProcessor](http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd...http://wiki.apache.org/solr/UpdateRequestProcessor) > > ## Thanks, Otis > > Sematext ::[http://sematext.com/::](http://sematext.com/::) Solr - Lucene - Nutch > Lucene ecosystem search ::[http://search-lucene.com/](http://search-lucene.com/) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 21, 2010, 4:54pm UTC](https://discuss.elastic.co/t/document-pre-processor/3119/4 "2010-07-21T16:54:23Z")
I responded to this, but it seems like you might have missed it, here is my answer again: There isn't really a hook point for this (i.e. get the content (json / xson) and munge it). It can be added, but I think that it make more sense to do it on the "client" side if one wish to do it, no? The fact that elasticsearch supports json like structures natively (with deep level objects) means that you can basically create that pre processor on your client side. Another option is to create your own mapper that handles the part of the data you want, but that also is I think more complicated than simply doing it on the client side. -shay.banon On Wed, Jul 21, 2010 at 6:52 PM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Hello, > > I looked at > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/index/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/index/) > and at > [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/engine/robin/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/engine/robin/) > , but couldn't find anything that sounds like Solr's > UpdateRequestProcessor. > > Am I not looking in the right places? > Is this feature missing but already on the roadmap? > > Thanks, > Otis > > On Jul 19, 4:59 pm, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote: > > > Hello, > > > > What is the best way to inject a custom document "processor" where one > > could manipulate the document and its fields before the document gets > > indexed? > > > > This is what Solr has, for example: > > [http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd...http://wiki.apache.org/solr/UpdateRequestProcessor](http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd...http://wiki.apache.org/solr/UpdateRequestProcessor) > > > ## Thanks, Otis > > > > Sematext ::[http://sematext.com/::](http://sematext.com/::) Solr - Lucene - Nutch > > Lucene ecosystem search ::[http://search-lucene.com/](http://search-lucene.com/) ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [July 21, 2010, 11:05pm UTC](https://discuss.elastic.co/t/document-pre-processor/3119/5 "2010-07-21T23:05:10Z")
Hi, On Jul 21, 12:54 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > I responded to this, but it seems like you might have missed it, here is my > answer again: Check [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/228cf9757b53b74e](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/228cf9757b53b74e) - no reply there. > There isn't really a hook point for this (i.e. get the content (json / xson) > and munge it). It can be added, but I think that it make more sense to do it > on the "client" side if one wish to do it, no? The fact that elasticsearch > supports json like structures natively (with deep level objects) means that > you can basically create that pre processor on your client side. Looks like that will have to be the route. > Another option is to create your own mapper that handles the part of the > data you want, but that also is I think more complicated than simply doing > it on the client side. The nice part about it being in ES is that different clients don't have to implement this or even be aware of documents being munged before indexing. Otis > On Wed, Jul 21, 2010 at 6:52 PM, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote: > > > Hello, > > > I looked at > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/index/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/index/) > > and at > > [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/engine/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/engine/)... > > , but couldn't find anything that sounds like Solr's > > UpdateRequestProcessor. > > > Am I not looking in the right places? > > Is this feature missing but already on the roadmap? > > > Thanks, > > Otis > > > On Jul 19, 4:59 pm, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote: > > > > > Hello, > > > > What is the best way to inject a custom document "processor" where one > > > could manipulate the document and its fields before the document gets > > > indexed? > > > > This is what Solr has, for example: > > > [http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd](http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd)... > > > > ## Thanks, Otis > > > > > > Sematext ::[http://sematext.com/::Solr](http://sematext.com/::Solr) - Lucene - Nutch > > > Lucene ecosystem search ::[http://search-lucene.com/](http://search-lucene.com/) ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [July 21, 2010, 11:13pm UTC](https://discuss.elastic.co/t/document-pre-processor/3119/6 "2010-07-21T23:13:40Z")
On Thu, Jul 22, 2010 at 1:05 AM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Hi, > > On Jul 21, 12:54 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > I responded to this, but it seems like you might have missed it, here is > > my > > answer again: > > Check > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/228cf9757b53b74e](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/228cf9757b53b74e) > > - no reply there. Interestingly, the reply is here: [http://elasticsearch-users.115913.n3.nabble.com/Document-pre-processor-tp979569p979702.html](http://elasticsearch-users.115913.n3.nabble.com/Document-pre-processor-tp979569p979702.html) > > There isn't really a hook point for this (i.e. get the content (json / > > xson) > > and munge it). It can be added, but I think that it make more sense to do > > it > > on the "client" side if one wish to do it, no? The fact that > > elasticsearch > > supports json like structures natively (with deep level objects) means > > that > > you can basically create that pre processor on your client side. > > Looks like that will have to be the route. > > > Another option is to create your own mapper that handles the part of the > > data you want, but that also is I think more complicated than simply > > doing > > it on the client side. > > The nice part about it being in ES is that different clients don't > have to implement this or even be aware of documents being munged > before indexing. > > Otis > > > On Wed, Jul 21, 2010 at 6:52 PM, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) > > wrote: > > > > > Hello, > > > > > I looked at > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/index/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/index/) > > > and at > > > [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/engine/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/engine/). > > > .. > > > , but couldn't find anything that sounds like Solr's > > > UpdateRequestProcessor. > > > > > Am I not looking in the right places? > > > Is this feature missing but already on the roadmap? > > > > > Thanks, > > > Otis > > > > > On Jul 19, 4:59 pm, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote: > > > > > > > Hello, > > > > > > What is the best way to inject a custom document "processor" where > > > > one > > > > could manipulate the document and its fields before the document gets > > > > indexed? > > > > > > This is what Solr has, for example: > > > > > [http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd](http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd). > > > .. > > > > > > ## Thanks, Otis > > > > > > > > Sematext ::[http://sematext.com/::Solr](http://sematext.com/::Solr) - Lucene - Nutch > > > > Lucene ecosystem search ::[http://search-lucene.com/](http://search-lucene.com/) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 22, 2010, 8:26am UTC](https://discuss.elastic.co/t/document-pre-processor/3119/7 "2010-07-22T08:26:08Z")
Very strange, looks like google groups is acting up... . Its a very valid point regarding the fact that you can have the same pre-processing power for different clients if it is built into elasticsearch. The way elasticsearch works with data is that the document (json) passed is never built into memory (as a Map of Maps/Lists/... for example), its pull parsed directly into a Lucene document (to same memory and speed up indexing). So, in order to have a hook point that changes it within elasticsearch, it will need to be built into memory, passed to the extension point to munge it, them converted back to an xcontent format (json) and then parse it again. -shay.banon On Thu, Jul 22, 2010 at 2:13 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > On Thu, Jul 22, 2010 at 1:05 AM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > > > Hi, > > > > On Jul 21, 12:54 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > I responded to this, but it seems like you might have missed it, here is > > > my > > > answer again: > > > > Check > > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/228cf9757b53b74e](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/228cf9757b53b74e) > > > > - no reply there. > > Interestingly, the reply is here: > [http://elasticsearch-users.115913.n3.nabble.com/Document-pre-processor-tp979569p979702.html](http://elasticsearch-users.115913.n3.nabble.com/Document-pre-processor-tp979569p979702.html) > > > > There isn't really a hook point for this (i.e. get the content (json / > > > xson) > > > and munge it). It can be added, but I think that it make more sense to > > > do it > > > on the "client" side if one wish to do it, no? The fact that > > > elasticsearch > > > supports json like structures natively (with deep level objects) means > > > that > > > you can basically create that pre processor on your client side. > > > > Looks like that will have to be the route. > > > > > Another option is to create your own mapper that handles the part of the > > > data you want, but that also is I think more complicated than simply > > > doing > > > it on the client side. > > > > The nice part about it being in ES is that different clients don't > > have to implement this or even be aware of documents being munged > > before indexing. > > > > Otis > > > > > On Wed, Jul 21, 2010 at 6:52 PM, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) > > > wrote: > > > > > > > Hello, > > > > > > > I looked at > > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/index/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/index/) > > > > and at > > > > [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/engine/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/engine/). > > > > .. > > > > , but couldn't find anything that sounds like Solr's > > > > UpdateRequestProcessor. > > > > > > > Am I not looking in the right places? > > > > Is this feature missing but already on the roadmap? > > > > > > > Thanks, > > > > Otis > > > > > > > On Jul 19, 4:59 pm, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote: > > > > > > > > > Hello, > > > > > > > > What is the best way to inject a custom document "processor" where > > > > > one > > > > > could manipulate the document and its fields before the document > > > > > gets > > > > > indexed? > > > > > > > > This is what Solr has, for example: > > > > > > > [http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd](http://search-lucene.com/jd/solr/org/apache/solr/update/processor/Upd). > > > > .. > > > > > > > > ## Thanks, Otis > > > > > > > > > > Sematext ::[http://sematext.com/::Solr](http://sematext.com/::Solr) - Lucene - Nutch > > > > > Lucene ecosystem search ::[http://search-lucene.com/](http://search-lucene.com/) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:21am UTC](https://discuss.elastic.co/t/document-pre-processor/3119/8 "2017-07-06T04:21:49Z")