# Extending the API (executing distributed actions) **URL:** \ **Category:** Elasticsearch\ **Created:** [December 7, 2012, 12:16am UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981 "2012-12-07T00:16:28Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [December 7, 2012, 12:16am UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/1 "2012-12-07T00:16:28Z")
I just built a plugin that provides a synonym token filter that loads terms from a database and not from a file. One additional feature that I would like to implement is the ability to reload the terms on demand. I figured adding code on the client side Java API would be too much work when the REST API will suffice for now. I created a REST action that receives the call, but it is at this point where I do not know how to continue. How can I execute a command on all nodes? Jumping into the code, I see that the Client interface provides methods for each concrete action (search, index, count, etc..) plus two generic execute() methods. First of all, what should the workflow be? Second, what do the specialized BroadcastOperation classes provide? Does the request execute on every node? I do not see where the magic happens. BroadcastOperation? For now, I have created my own Action, Request, Response and RequestBuilder stubs, and everything compiles. What is confusing is the I create a new Action, which contains a method that creates a new RequestBuilder and is used as a param to Client#execute(). This class in turn has a doExecute() method, which also calls out to the Client. Am I missing something? Cheers, Ivan -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 7, 2012, 2:10am UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/2 "2012-12-07T02:10:48Z")
Just a quick overview: - there are action levels, an action can be executed on node level, on indices level, and on shard level - there are action classes (broadcast, master, node, replication, single custom, single shard etc.) - there are actions for ingesting data (without Lucene code), for queries (with Lucene query builders) and for adminstration (cluster and indices) - there is, for each action, a request/response class, with a request builder class, a Transport...Action, and shard level request/response - in general, the control flow is moving the request to the indices level, from there to the shard level, then responses are created and transported back - the heavy lifting is done by a Transport...Action where different layers are coordinated - such an action can be additionally exposed by the HTTP REST API As there are so much ES actions, it might be easier to study plugins with actions first, how they add their action to ES. Don't get confused by the execute() / doExecute() methods, they just organize the client API access, how clients should execute from a number of selected actions in a registry. Each API call is implemented by a listener call style and a prepare/execute style. I hope this helps, feel free to ask for more details, Jörg -- ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [December 7, 2012, 6:49am UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/3 "2012-12-07T06:49:23Z")
Thanks Jörg. Much of what you have told me I already know! One thing that I did miss is that each Action has a corresponding TransportAction that is binded in the ActionModule. That important step was missing to me and not obvious. What is still missing is how to differentiate between action levels. I have written other plugins, but I have always interacted with the "specific" methods defined in Client, never the generic execute(). Client.get(), .index(), etc... all perform the actions at the necessary levels. Most plugins work in this fashion. In the end, there are far too many classes for such a simple feature. I am just going to implement a timer thread that updates the synonyms at a specific interval. Cheers, Ivan On Thu, Dec 6, 2012 at 6:10 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > - there is, for each action, a request/response class, with a request > builder class, a Transport...Action, and shard level request/response -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 7, 2012, 7:50am UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/4 "2012-12-07T07:50:38Z")
Hi Ivan, only for the sake of completeness, fetching data from external sources on a regular basis is a common task for river plugins. So you might have a look into the JDBC river plugin as well. Cheers, Jörg On Friday, December 7, 2012 7:49:23 AM UTC+1, Ivan Brusic wrote: > Thanks Jörg. > > Much of what you have told me I already know! One thing that I did miss is > that each Action has a corresponding TransportAction that is binded in the > ActionModule. That important step was missing to me and not obvious. > > What is still missing is how to differentiate between action levels. I > have written other plugins, but I have always interacted with the > "specific" methods defined in Client, never the generic execute(). > Client.get(), .index(), etc... all perform the actions at the necessary > levels. Most plugins work in this fashion. > > In the end, there are far too many classes for such a simple feature. I am > just going to implement a timer thread that updates the synonyms at a > specific interval. > > Cheers, > > Ivan > > On Thu, Dec 6, 2012 at 6:10 PM, Jörg Prante \<[joerg...@gmail.com](mailto:joerg...@gmail.com)\ > > > wrote: > > > - there is, for each action, a request/response class, with a request > > builder class, a Transport...Action, and shard level request/response -- ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [December 7, 2012, 5:20pm UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/5 "2012-12-07T17:20:53Z")
Thanks again Jörg. I wrote my first river plugin almost two years ago, so I know quite well how they work. 🙂 Every river plugin eventually calls client.index(...) and uses the existing Action/Request/Response classes, so all the heavy lifting is already done. Besides, I am not updating an index, just some in-memory list (node-level). As you mentioned, most of the work is done by the TransportAction classes. The decision between using the appropriate service (cluster, transport, index, script) lies there. Since most of the configuration happens via DI, I completely missed the TransportAction classes. But like I mentioned before, it is far too much code for a simple refresh. A timer thread is not elegant, but it's only a few lines of code! I was hoping for a less kludgy solution so I could open-source the filter. Then again, the current file-based synonym/stopwords/keyword-marker/etc filters are not refreshed either, so the problem is not a major one. But I would still like to understand the class interactions, so I could perhaps write a HOWTO. Cheers, Ivan On Thu, Dec 6, 2012 at 11:50 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Hi Ivan, > > only for the sake of completeness, fetching data from external sources on > a regular basis is a common task for river plugins. So you might have a > look into the JDBC river plugin as well. > > Cheers, > > Jörg > > On Friday, December 7, 2012 7:49:23 AM UTC+1, Ivan Brusic wrote: > > > Thanks Jörg. > > > > Much of what you have told me I already know! One thing that I did miss > > is that each Action has a corresponding TransportAction that is binded in > > the ActionModule. That important step was missing to me and not obvious. > > > > What is still missing is how to differentiate between action levels. I > > have written other plugins, but I have always interacted with the > > "specific" methods defined in Client, never the generic execute(). > > Client.get(), .index(), etc... all perform the actions at the necessary > > levels. Most plugins work in this fashion. > > > > In the end, there are far too many classes for such a simple feature. I > > am just going to implement a timer thread that updates the synonyms at a > > specific interval. > > > > Cheers, > > > > Ivan > > > > On Thu, Dec 6, 2012 at 6:10 PM, Jörg Prante [joerg...@gmail.com](mailto:joerg...@gmail.com) wrote: > > > > > - there is, for each action, a request/response class, with a request > > > builder class, a Transport...Action, and shard level request/response > > > > -- -- ---
**Author:** ![Bruce\_Ritchie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruce_ritchie/32/9370_2.png) [@Bruce\_Ritchie](https://discuss.elastic.co/u/Bruce_Ritchie)\ **Post date:** [December 13, 2012, 3:13pm UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/6 "2012-12-13T15:13:26Z")
Ivan, Do you have any plans to open source this plugin? I was about to write a plugin with the same functionality and I suspect it's something that may be a fairly common request. All the best, Bruce -- ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [December 13, 2012, 5:35pm UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/7 "2012-12-13T17:35:53Z")
The plugin is quite simple, but I guess if there is a demand I could open source it. There is some custom code needed for my company (database connectivity) and it needs to be refactored. My ultimate goal is to abstract the code to support all file-based token filters (eg: stopwords). My plan was to work on it a bit more during the Christmas slowdown, but if you need something sooner, I can help you out. It basically comes down to re-working SynonymTokenFilterFactory. -- Ivan On Thu, Dec 13, 2012 at 7:13 AM, Bruce Ritchie [bruce.ritchie@gmail.com](mailto:bruce.ritchie@gmail.com)wrote: > Ivan, > > Do you have any plans to open source this plugin? I was about to write a > plugin with the same functionality and I suspect it's something that may be > a fairly common request. > > All the best, > > Bruce > > -- -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 13, 2012, 6:02pm UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/8 "2012-12-13T18:02:50Z")
Maybe the SKOS pluging can also give some insight. It is kind of a synonym expander with controlled vocabulary from an external semantic web resource. If you can wrap your synonyms into SKOS, this plugin might be useful. > **[GitHub - jprante/elasticsearch-analysis-skos: SKOS analysis for Elasticsearch](https://github.com/jprante/elasticsearch-analysis-skos)** > > SKOS analysis for Elasticsearch. Contribute to jprante/elasticsearch-analysis-skos development by creating an account on GitHub. Cheers, Jörg On Thursday, December 13, 2012 6:35:53 PM UTC+1, Ivan Brusic wrote: > The plugin is quite simple, but I guess if there is a demand I could open > source it. There is some custom code needed for my company > (database connectivity) and it needs to be refactored. My ultimate goal is > to abstract the code to support all file-based token filters (eg: > stopwords). > > My plan was to work on it a bit more during the Christmas slowdown, but if > you need something sooner, I can help you out. It basically comes down to > re-working SynonymTokenFilterFactory. > > -- > Ivan > > On Thu, Dec 13, 2012 at 7:13 AM, Bruce Ritchie \<[bruce....@gmail.com](mailto:bruce....@gmail.com)\ > > > wrote: > > > Ivan, > > > > Do you have any plans to open source this plugin? I was about to write a > > plugin with the same functionality and I suspect it's something that may be > > a fairly common request. > > > > All the best, > > > > Bruce > > > > -- -- ---
**Author:** ![Bruce\_Ritchie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruce_ritchie/32/9370_2.png) [@Bruce\_Ritchie](https://discuss.elastic.co/u/Bruce_Ritchie)\ **Post date:** [December 13, 2012, 6:18pm UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/9 "2012-12-13T18:18:34Z")
No immediately need, no. My timeline is 1st quarter next year. Bruce On Thu, Dec 13, 2012 at 12:35 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > The plugin is quite simple, but I guess if there is a demand I could open > source it. There is some custom code needed for my company > (database connectivity) and it needs to be refactored. My ultimate goal is > to abstract the code to support all file-based token filters (eg: > stopwords). > > My plan was to work on it a bit more during the Christmas slowdown, but if > you need something sooner, I can help you out. It basically comes down to > re-working SynonymTokenFilterFactory. > > -- > Ivan > > On Thu, Dec 13, 2012 at 7:13 AM, Bruce Ritchie [bruce.ritchie@gmail.com](mailto:bruce.ritchie@gmail.com)wrote: > > > Ivan, > > > > Do you have any plans to open source this plugin? I was about to write a > > plugin with the same functionality and I suspect it's something that may be > > a fairly common request. > > > > All the best, > > > > Bruce > > > > -- > > -- -- Support me in the Ride to Conquer Cancer @ [http://www.conquercancer.ca/site/TR/Events/Toronto2013?px=3106212&pg=personal&fr\_id=1431](http://www.conquercancer.ca/site/TR/Events/Toronto2013?px=3106212&pg=personal&fr_id=1431) -- ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [December 13, 2012, 7:57pm UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/10 "2012-12-13T19:57:29Z")
My goal was to load synonyms from a database instead of a file, and not so much what to load, but you made me think of another point. If the Maven-Plugin project is a success (and it will be), then your plugin could simply depend on the db-loader plugin I created, although I would not know how to present skos files in a database. BTW, I might steal^H^H^H^H^H borrow your SqlService class since mine is not generic (custom datasources). -- Ivan On Thu, Dec 13, 2012 at 10:02 AM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > synonym expander with controlled vocabulary from an -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:59am UTC](https://discuss.elastic.co/t/extending-the-api-executing-distributed-actions/9981/11 "2017-07-06T02:59:55Z")