# Pre-commit data transformation plugin **URL:** \ **Category:** Elasticsearch\ **Created:** [August 26, 2011, 7:23pm UTC](https://discuss.elastic.co/t/pre-commit-data-transformation-plugin/5260 "2011-08-26T19:23:51Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![David\_Richardson](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@David\_Richardson](https://discuss.elastic.co/u/David_Richardson)\ **Post date:** [August 26, 2011, 7:23pm UTC](https://discuss.elastic.co/t/pre-commit-data-transformation-plugin/5260/1 "2011-08-26T19:23:51Z")
We need the ability to transform a document prior to commit - in this case to create a structure to retain data revisions. Without using middleware. I was pointed to the ability to write preIndex hooks on #elasticsearch and given the Percolate module as an example. I still have a number of specific questions about such an approach: - Is this the best template for understanding such functionality? - Has anyone done something similar and published the code? - How would such code be packaged? Percolate doesn't appear to be in 'plugin' format (if that's even the correct terminology). I do see a plugin class when browsing the source, but Percolate doesn't appear to subclass this. - Is there a formal structure for extensions to ES, even if unpublished? I presume this is the case, and that plugins follow it. - How are external extensions registered for use? I presume the Plugins folder is traversed and all .jars are loaded? - I recall seeing reference to Javascript for use elsewhere in ES - I presume Rhino is being used. Is it then available for general use (i.e. as the language to perform transformations pre-commit)? What version are we using? - Modules vs Plugins. What's the difference? - The process of transforming the document will be a synchronous action. Does this have non-obvious implications? Is there a timeout management mechanism inherent in all request processing that would implicitly supervise this? Suggestions/help much appreciated. david ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 29, 2011, 6:04pm UTC](https://discuss.elastic.co/t/pre-commit-data-transformation-plugin/5260/2 "2011-08-29T18:04:13Z")
Answers below, note though, with transformation, what you want to do is perform the transformation onces, and then use the transformed data when executing the index operation on both primary shard and replica shards. In genreal though, I am not a fan of having such functionality in elasticsearch, and think that it should be done outside of elasticsearch. What I am concerned about is that this type of features tend to get misused and abused very easily. Obviously, I won't stand in the way of huge demand for it. On Fri, Aug 26, 2011 at 10:23 PM, David Richardson \< [david.richardson@enquora.com](mailto:david.richardson@enquora.com)\> wrote: > We need the ability to transform a document prior to commit - in this case > to create a structure to retain data revisions. Without using middleware. > I was pointed to the ability to write preIndex hooks on #elasticsearch and > given the Percolate module as an example. > I still have a number of specific questions about such an approach: > > - Is this the best template for understanding such functionality? > > I don't think it fits this case, because you want to perform the > "transformation" once. There isn't a good extension point in elasticsearch > to do that now. > - Has anyone done something similar and published the code? > - How would such code be packaged? Percolate doesn't appear to be in > 'plugin' format (if that's even the correct terminology). I do see a plugin > class when browsing the source, but Percolate doesn't appear to subclass > this. > > Check the other plugins in elasticsearch. Basically, they are zip files > that end up being extracted under the plugins directory. All jar / classes > files within that plugin directory under plugins are loaded automatically. > - Is there a formal structure for extensions to ES, even if > unpublished? I presume this is the case, and that plugins follow it. > > There is a mini format one. Check some of the plugins under elasticsearch > itself. As for what can be extended, that depends on the extension points / > guice injection options. > - How are external extensions registered for use? I presume the Plugins > folder is traversed and all .jars are loaded? > > Yes, it loads all the jar files, and calls the class listed in the > es-plugin.properties file. > - I recall seeing reference to Javascript for use elsewhere in ES - I > presume Rhino is being used. Is it then available for general use (i.e. as > the language to perform transformations pre-commit)? What version are we > using? > > The rhino version used is 1.7R2. The whole Scripting aspect is available > for general use using the ScriptingService, and its used in different > places. > - Modules vs Plugins. What's the difference? > > Modules are guice concepts. A plugin can have many modules. > - The process of transforming the document will be a synchronous > action. Does this have non-obvious implications? Is there a timeout > management mechanism inherent in all request processing that would > implicitly supervise this? > > Depends on where its plugged in, but, it will simply wait for it to finish. > Suggestions/help much appreciated. > david ---
**Author:** ![David\_Richardson](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@David\_Richardson](https://discuss.elastic.co/u/David_Richardson)\ **Post date:** [August 29, 2011, 10:44pm UTC](https://discuss.elastic.co/t/pre-commit-data-transformation-plugin/5260/3 "2011-08-29T22:44:10Z")
Thks Shay. There'll be more questions but that helps. david ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:55am UTC](https://discuss.elastic.co/t/pre-commit-data-transformation-plugin/5260/4 "2017-07-06T03:55:38Z")
Closed on July 6, 2017, 3:55am UTC