# Custom components **URL:** \ **Category:** Elasticsearch\ **Created:** [July 12, 2010, 8:09am UTC](https://discuss.elastic.co/t/custom-components/3088 "2010-07-12T08:09:38Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Tomislav\_Poljak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomislav_poljak/32/1177_2.png) [@Tomislav\_Poljak](https://discuss.elastic.co/u/Tomislav_Poljak)\ **Post date:** [July 12, 2010, 8:09am UTC](https://discuss.elastic.co/t/custom-components/3088/1 "2010-07-12T08:09:38Z")
Hi, I've been reviewing and analyzing Elastic Search for one project and from what I've seen so far it seems really great! I have a question regarding custom (component) development inside Elastic Search to satisfy specific requirements. First, we need to be able to execute batch indexing (using ES Java API) on HTTP request (location of folder containing data for indexing would be a parameter in request). What would be the best (easiest) way to implement this (by extending ...)? Second, we need Auditing (what and when was changed in index) or maybe even all versions of 'original json document' stored. I know an update of a Lucene document is actually an insert (new) and delete (old) against Lucene index. Also, probably the simplest way of having all versions of 'original json document' stored (in a \_source field) would be to simple insert a new document and not delete the old document version (by using custom non-ES ID as a non-unique docID and tagging a newly inserted document with ACTIVE tag/flag). Of course, this approach would create multiple instances for each document (after each update) in index, so index would be much bigger and harder to manage + this would be bad for search performance. Here is my question: would it be possible to implement this non-delete-on-update logic in a custom gateway? For active indexing/search everything would work as usual and active index would be replicated through 'standard gateway' as usual, but there would be an 'Audit/Historical gateway' which would only insert document on update (and not delete the old versions). I understand that this requires different unique ID policy on active vs. 'Audit/Historical gateway' index. Maybe it would be possible to re-use ES's transaction log to 'replay' operations on other/'different unique ID policy' index? Also, if possible, not all fields from active index need to be replicated in Audit/Historical 'all versions stored' index. Is there a simple way of implementing 'all versions of document' history in Elastic Search? Tomislav ---
**Author:** ![Tomislav\_Poljak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomislav_poljak/32/1177_2.png) [@Tomislav\_Poljak](https://discuss.elastic.co/u/Tomislav_Poljak)\ **Post date:** [July 12, 2010, 8:33am UTC](https://discuss.elastic.co/t/custom-components/3088/2 "2010-07-12T08:33:42Z")
Hi, just one clarification on the first question: question is not about how to use ES Java API for batch indexing (I've found info/code on this in documentation). I'm interested in what needs to be done (extended) to add custom 'web' http/json command/component which has an input paramas (http/json) and outputs some (simple) response. Tomislav On Mon, 2010-07-12 at 10:09 +0200, Tomislav Poljak wrote: > Hi, > I've been reviewing and analyzing Elastic Search for one project and > from what I've seen so far it seems really great! > > I have a question regarding custom (component) development inside > Elastic Search to satisfy specific requirements. > > First, we need to be able to execute batch indexing (using ES Java API) > on HTTP request (location of folder containing data for indexing would > be a parameter in request). What would be the best (easiest) way to > implement this (by extending ...)? > > Second, we need Auditing (what and when was changed in index) or maybe > even all versions of 'original json document' stored. I know an update > of a Lucene document is actually an insert (new) and delete (old) > against Lucene index. Also, probably the simplest way of having all > versions of 'original json document' stored (in a \_source field) would > be to simple insert a new document and not delete the old document > version (by using custom non-ES ID as a non-unique docID and tagging a > newly inserted document with ACTIVE tag/flag). > > Of course, this approach would create multiple instances for each > document (after each update) in index, so index would be much bigger and > harder to manage + this would be bad for search performance. > > Here is my question: would it be possible to implement this > non-delete-on-update logic in a custom gateway? > > For active indexing/search everything would work as usual and active > index would be replicated through 'standard gateway' as usual, but there > would be an 'Audit/Historical gateway' which would only insert document > on update (and not delete the old versions). I understand that this > requires different unique ID policy on active vs. 'Audit/Historical > gateway' index. > > Maybe it would be possible to re-use ES's transaction log to 'replay' > operations on other/'different unique ID policy' index? > > Also, if possible, not all fields from active index need to be > replicated in Audit/Historical 'all versions stored' index. > > Is there a simple way of implementing 'all versions of document' history > in Elastic Search? > > Tomislav ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 12, 2010, 6:06pm UTC](https://discuss.elastic.co/t/custom-components/3088/3 "2010-07-12T18:06:10Z")
Hi, Regarding the first question, I am not sure why you would want to extend elasticsearch to do it. You can create a simple program that uses elasticsearch API and exposes a REST endpoint to do what you want. This is preferable since you are more isolated form elasticsearch internal changes. Regarding the second question, again, I think you are trying to solve this too much "inside" elasticsearch. Where do you plan to store your data? Maybe you can do your versioning there, and always keep only the latest docs in elasticsearch. -shay.banon On Mon, Jul 12, 2010 at 11:33 AM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com) wrote: > Hi, > just one clarification on the first question: > > question is not about how to use ES Java API for batch indexing (I've > found info/code on this in documentation). I'm interested in what needs > to be done (extended) to add custom 'web' http/json command/component > which has an input paramas (http/json) and outputs some (simple) > response. > > Tomislav > > On Mon, 2010-07-12 at 10:09 +0200, Tomislav Poljak wrote: > > > Hi, > > I've been reviewing and analyzing Elastic Search for one project and > > from what I've seen so far it seems really great! > > > > I have a question regarding custom (component) development inside > > Elastic Search to satisfy specific requirements. > > > > First, we need to be able to execute batch indexing (using ES Java API) > > on HTTP request (location of folder containing data for indexing would > > be a parameter in request). What would be the best (easiest) way to > > implement this (by extending ...)? > > > > Second, we need Auditing (what and when was changed in index) or maybe > > even all versions of 'original json document' stored. I know an update > > of a Lucene document is actually an insert (new) and delete (old) > > against Lucene index. Also, probably the simplest way of having all > > versions of 'original json document' stored (in a \_source field) would > > be to simple insert a new document and not delete the old document > > version (by using custom non-ES ID as a non-unique docID and tagging a > > newly inserted document with ACTIVE tag/flag). > > > > Of course, this approach would create multiple instances for each > > document (after each update) in index, so index would be much bigger and > > harder to manage + this would be bad for search performance. > > > > Here is my question: would it be possible to implement this > > non-delete-on-update logic in a custom gateway? > > > > For active indexing/search everything would work as usual and active > > index would be replicated through 'standard gateway' as usual, but there > > would be an 'Audit/Historical gateway' which would only insert document > > on update (and not delete the old versions). I understand that this > > requires different unique ID policy on active vs. 'Audit/Historical > > gateway' index. > > > > Maybe it would be possible to re-use ES's transaction log to 'replay' > > operations on other/'different unique ID policy' index? > > > > Also, if possible, not all fields from active index need to be > > replicated in Audit/Historical 'all versions stored' index. > > > > Is there a simple way of implementing 'all versions of document' history > > in Elastic Search? > > > > Tomislav ---
**Author:** ![Tomislav\_Poljak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomislav_poljak/32/1177_2.png) [@Tomislav\_Poljak](https://discuss.elastic.co/u/Tomislav_Poljak)\ **Post date:** [July 13, 2010, 10:49am UTC](https://discuss.elastic.co/t/custom-components/3088/4 "2010-07-13T10:49:36Z")
Hi, On Mon, 2010-07-12 at 21:06 +0300, Shay Banon wrote: > Hi, > > Regarding the first question, I am not sure why you would want to > extend elasticsearch to do it. You can create a simple program that > uses elasticsearch API and exposes a REST endpoint to do what you > want. This is preferable since you are more isolated form > elasticsearch internal changes. Yes, this is way my initial idea to implement servlet and deploy it on tomcat/jetty (since this bulk/batch indexing application needs to be 'executed' over http). This servlet would be using ES java API for bulk indexing and bulk updates. But, since ES already runs it's own java server (servlet container of some kind?), I thought maybe it would be simpler to 'integrate' this custom servlet in ES web server/container. Is this possible or would you recommend separate tomcat/jetty with 'bulk indexer' servlet? > Regarding the second question, again, I think you are trying to > solve this too much "inside" elasticsearch. Where do you plan to store > your data? Data will be replicated through gateway on share disk (NAS). But, we would have 2 index versions: one active with 'single' document instance (used for searching) and one 'historical' with multiple document versions (each update on document creates a new version, this version would be used for 'historical' search). > Maybe you can do your versioning there, and always keep only the > latest docs in elasticsearch. Yes, I think you are right. I guess I can re-post each document which is posted to 'active ES' for indexing/update to 'historical ES' instance with different unique key policy (where update will be treated as new document insert, because of different unique key defined in type mapping) Tomislav > -shay.banon > > On Mon, Jul 12, 2010 at 11:33 AM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com) > wrote: > Hi, > just one clarification on the first question: > > ``` > question is not about how to use ES Java API for batch > indexing (I've > found info/code on this in documentation). I'm interested in > what needs > to be done (extended) to add custom 'web' http/json > command/component > which has an input paramas (http/json) and outputs some > (simple) > response. > > Tomislav > > > > > On Mon, 2010-07-12 at 10:09 +0200, Tomislav Poljak wrote: > > Hi, > > I've been reviewing and analyzing Elastic Search for one > project and > > from what I've seen so far it seems really great! > > > > I have a question regarding custom (component) development > inside > > Elastic Search to satisfy specific requirements. > > > > First, we need to be able to execute batch indexing (using > ES Java API) > > on HTTP request (location of folder containing data for > indexing would > > be a parameter in request). What would be the best (easiest) > way to > > implement this (by extending ...)? > > > > Second, we need Auditing (what and when was changed in > index) or maybe > > even all versions of 'original json document' stored. I know > an update > > of a Lucene document is actually an insert (new) and delete > (old) > > against Lucene index. Also, probably the simplest way of > having all > > versions of 'original json document' stored (in a _source > field) would > > be to simple insert a new document and not delete the old > document > > version (by using custom non-ES ID as a non-unique docID and > tagging a > > newly inserted document with ACTIVE tag/flag). > > > > Of course, this approach would create multiple instances for > each > > document (after each update) in index, so index would be > much bigger and > > harder to manage + this would be bad for search performance. > > > > Here is my question: would it be possible to implement this > > non-delete-on-update logic in a custom gateway? > > > > For active indexing/search everything would work as usual > and active > > index would be replicated through 'standard gateway' as > usual, but there > > would be an 'Audit/Historical gateway' which would only > insert document > > on update (and not delete the old versions). I understand > that this > > requires different unique ID policy on active vs. > 'Audit/Historical > > gateway' index. > > > > Maybe it would be possible to re-use ES's transaction log to > 'replay' > > operations on other/'different unique ID policy' index? > > > > Also, if possible, not all fields from active index need to > be > > replicated in Audit/Historical 'all versions stored' index. > > > > Is there a simple way of implementing 'all versions of > document' history > > in Elastic Search? > > > > > > Tomislav > > > > > > > > ``` ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 13, 2010, 12:05pm UTC](https://discuss.elastic.co/t/custom-components/3088/5 "2010-07-13T12:05:52Z")
Hey, answers below: On Tue, Jul 13, 2010 at 1:49 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com) wrote: > Hi, > > On Mon, 2010-07-12 at 21:06 +0300, Shay Banon wrote: > > > Hi, > > > > Regarding the first question, I am not sure why you would want to > > extend elasticsearch to do it. You can create a simple program that > > uses elasticsearch API and exposes a REST endpoint to do what you > > want. This is preferable since you are more isolated form > > elasticsearch internal changes. > > Yes, this is way my initial idea to implement servlet and deploy it on > tomcat/jetty (since this bulk/batch indexing application needs to be > 'executed' over http). This servlet would be using ES java API for bulk > indexing and bulk updates. > > But, since ES already runs it's own java server (servlet container of > some kind?), I thought maybe it would be simpler to 'integrate' this > custom servlet in ES web server/container. > > Is this possible or would you recommend separate tomcat/jetty with 'bulk > indexer' servlet? I would recommend doing it externally to elasticsearch. ES does not use jetty or tomcat, since implementing non blocking / async IO with operations is not very well implemented in them. It has a nice abstraction to plug your own REST services and so on, but its not aimed at hosting your code except for extension to ES. > > Regarding the second question, again, I think you are trying to > > solve this too much "inside" elasticsearch. Where do you plan to store > > your data? > > Data will be replicated through gateway on share disk (NAS). But, we > would have 2 index versions: one active with 'single' document instance > (used for searching) and one 'historical' with multiple document > versions (each update on document creates a new version, this version > would be used for 'historical' search). > > > Maybe you can do your versioning there, and always keep only the > > latest docs in elasticsearch. > > Yes, I think you are right. I guess I can re-post each document which is > posted to 'active ES' for indexing/update to 'historical ES' instance > with different unique key policy (where update will be treated as new > document insert, because of different unique key defined in type > mapping) Make sense. > Tomislav > > > -shay.banon > > > > On Mon, Jul 12, 2010 at 11:33 AM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com) > > wrote: > > Hi, > > just one clarification on the first question: > > > > ``` > > question is not about how to use ES Java API for batch > > indexing (I've > > found info/code on this in documentation). I'm interested in > > what needs > > to be done (extended) to add custom 'web' http/json > > command/component > > which has an input paramas (http/json) and outputs some > > (simple) > > response. > > > > Tomislav > > > > On Mon, 2010-07-12 at 10:09 +0200, Tomislav Poljak wrote: > > > Hi, > > > I've been reviewing and analyzing Elastic Search for one > > project and > > > from what I've seen so far it seems really great! > > > > > > I have a question regarding custom (component) development > > inside > > > Elastic Search to satisfy specific requirements. > > > > > > First, we need to be able to execute batch indexing (using > > ES Java API) > > > on HTTP request (location of folder containing data for > > indexing would > > > be a parameter in request). What would be the best (easiest) > > way to > > > implement this (by extending ...)? > > > > > > Second, we need Auditing (what and when was changed in > > index) or maybe > > > even all versions of 'original json document' stored. I know > > an update > > > of a Lucene document is actually an insert (new) and delete > > (old) > > > against Lucene index. Also, probably the simplest way of > > having all > > > versions of 'original json document' stored (in a _source > > field) would > > > be to simple insert a new document and not delete the old > > document > > > version (by using custom non-ES ID as a non-unique docID and > > tagging a > > > newly inserted document with ACTIVE tag/flag). > > > > > > Of course, this approach would create multiple instances for > > each > > > document (after each update) in index, so index would be > > much bigger and > > > harder to manage + this would be bad for search performance. > > > > > > Here is my question: would it be possible to implement this > > > non-delete-on-update logic in a custom gateway? > > > > > > For active indexing/search everything would work as usual > > and active > > > index would be replicated through 'standard gateway' as > > usual, but there > > > would be an 'Audit/Historical gateway' which would only > > insert document > > > on update (and not delete the old versions). I understand > > that this > > > requires different unique ID policy on active vs. > > 'Audit/Historical > > > gateway' index. > > > > > > Maybe it would be possible to re-use ES's transaction log to > > 'replay' > > > operations on other/'different unique ID policy' index? > > > > > > Also, if possible, not all fields from active index need to > > be > > > replicated in Audit/Historical 'all versions stored' index. > > > > > > Is there a simple way of implementing 'all versions of > > document' history > > > in Elastic Search? > > > > > > > > > Tomislav > > > > > > > > > > > > > ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:22am UTC](https://discuss.elastic.co/t/custom-components/3088/6 "2017-07-06T04:22:20Z")