# Node plugin not propagating requests **URL:** \ **Category:** Elasticsearch\ **Created:** [September 18, 2013, 8:06pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681 "2013-09-18T20:06:21Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [September 18, 2013, 8:06pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/1 "2013-09-18T20:06:21Z")
Creating a new plugin that works on the node (JVM) level. The node that receives a request executes the request perfectly, but the other nodes do not get the request. In my TransportNodesOperationAction subclass, I implemented the two methods: protected abstract NodeRequest newNodeRequest(); protected abstract NodeRequest newNodeRequest(String nodeId, Request request); Execute the action via: client.admin().cluster().execute(MyAction.INSTANCE, request, new ActionListener() { ...}) When calling the new ClusterAction, the first node executes the newNodeRequest with the Request as a param. The other node gets the empty call. The correct method call occurs inside the AsyncAction.start() method. From my logging, my NodesOperationRequestBuilder never execute its doExecute() method. ``` @Override protected void doExecute(ActionListener listener) { ``` ((ClusterAdminClient) client).execute(MyrAction.INSTANCE, request, listener); } Not sure what I have missed. All classes subclass from the NodesOperation heirarchy (except for the Action which subclasses ClusterAction). executor() returns ThreadPool.Names.MANAGEMENT. Cheers, Ivan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 18, 2013, 8:20pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/2 "2013-09-18T20:20:52Z")
Can you share more than just code fragments, it's hard to find out what is going on. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [September 18, 2013, 9:50pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/3 "2013-09-18T21:50:52Z")
There are at least 7 different public classes needed for the plugin, and I don't have them in any public repo. Any specific class? On Wed, Sep 18, 2013 at 1:20 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > Can you share more than just code fragments, it's hard to find out what is > going on. > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [September 18, 2013, 10:45pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/4 "2013-09-18T22:45:18Z")
A few trimmed files: [https://gist.github.com/brusic/05c5af4c12357e8e49e5](https://gist.github.com/brusic/05c5af4c12357e8e49e5) On Wed, Sep 18, 2013 at 2:50 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > There are at least 7 different public classes needed for the plugin, and I > don't have them in any public repo. Any specific class? > > On Wed, Sep 18, 2013 at 1:20 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > Can you share more than just code fragments, it's hard to find out what > > is going on. > > > > Jörg > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [September 19, 2013, 5:38pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/5 "2013-09-19T17:38:06Z")
Is there anyone beside Jörg that knows elasticsearch internals? On Wed, Sep 18, 2013 at 3:45 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > A few trimmed files: [https://gist.github.com/brusic/05c5af4c12357e8e49e5](https://gist.github.com/brusic/05c5af4c12357e8e49e5) > > On Wed, Sep 18, 2013 at 2:50 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > There are at least 7 different public classes needed for the plugin, and > > I don't have them in any public repo. Any specific class? > > > > On Wed, Sep 18, 2013 at 1:20 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > Can you share more than just code fragments, it's hard to find out what > > > is going on. > > > > > > Jörg > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [September 19, 2013, 9:32pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/6 "2013-09-19T21:32:39Z")
Serialization. Duh. I made some assumptions that simply were not true. I will write-up my findings for anyone interested in writing node-level plugins in the future. -- Ivan On Thu, Sep 19, 2013 at 10:38 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > Is there anyone beside Jörg that knows elasticsearch internals? > > On Wed, Sep 18, 2013 at 3:45 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > A few trimmed files: [https://gist.github.com/brusic/05c5af4c12357e8e49e5](https://gist.github.com/brusic/05c5af4c12357e8e49e5) > > > > On Wed, Sep 18, 2013 at 2:50 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > > > There are at least 7 different public classes needed for the plugin, and > > > I don't have them in any public repo. Any specific class? > > > > > > On Wed, Sep 18, 2013 at 1:20 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > > > Can you share more than just code fragments, it's hard to find out what > > > > is going on. > > > > > > > > Jörg > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [September 20, 2013, 7:00am UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/7 "2013-09-20T07:00:28Z")
> I will write-up my findings for anyone interested in writing node-level > plugins in the future. +1 btw, my colleague Vlastimil implemented plugin that might be worth looking at [1]. I am not too familiar with its internals but IIRC he had to go down the similar road. [1] [GitHub - searchisko/elasticsearch-river-jira: JIRA River Plugin for Elasticsearch](https://github.com/jbossorg/elasticsearch-river-jira) Regards, Lukas -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [September 20, 2013, 4:31pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/8 "2013-09-20T16:31:04Z")
Thanks Lukas. My first plugin was a river plugin and that was three years ago. River plugins are far easier since all the node communication is handled by the RiverModule. Since then I have created other plugins (river and analysis), but nothing at the node level. Basically all data shared between nodes is serialized in the readFrom/writeTo methods of the NodeOperationRequest object. Since this class takes a NodesOperationRequest (note the different classname) as an arg in the constructor, I assumed that it would correctly serialize the request. However this obviously was not the case, which makes sense in hindsight. How can my subclass be automatically serialized if I added new fields? There are two different request classes and two different response classes, so it was hard to spot the issue without stepping back and looking at the class hierarchy as a whole. -- Ivan On Fri, Sep 20, 2013 at 12:00 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > I will write-up my findings for anyone interested in writing node-level > > > plugins in the future. > > +1 > > btw, my colleague Vlastimil implemented plugin that might be worth looking > at [1]. I am not too familiar with its internals but IIRC he had to go down > the similar road. > > [1] [GitHub - searchisko/elasticsearch-river-jira: JIRA River Plugin for Elasticsearch](https://github.com/jbossorg/elasticsearch-river-jira) > > Regards, > Lukas > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [September 24, 2013, 8:37pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/9 "2013-09-24T20:37:54Z")
Hi Ivan, yes I agree, but let me explain why I pointed out the JIRA river. Rivers are relatively easy that is true - but only as long as you are fine with its basic contract, i.e. start() and close() operations. Once you want more fine grained control of your river then it gets more complicated. In our rivers: JIRA river [1] and Remote river [2] (both are basically similar, the former is just more specific to JIRA as a data source) we also wanted to provide REST API for management of individual river instances and reporting status [3]. Extending REST API is again simple - by registering new actions on RestModule. But the question is what should happen when the REST request is received on the node which is not running the river singleton. Now, from what I have seen there are basically two approaches how ppl do go about this. The first one is to store client action as a document into cluster (for example into \_river index) and have the river check for such documents in periodic intervals. The river uses the search API to learn if there was any request to execute some specific action. The second way (which we used) is a bit different. In nutshell it includes full propagation from RestRequest to ClusterAction (and this includes NodeOperationRequest/Response serialization as well). I think this provides better options in terms of river control and reporting the status from river. May be there are better ways of doing this but as of now we do not know about them. Also I am not aware of any other plugin that would do it in similar way. It took Vlastimil quite some work to get there, one of the biggest issues was missing Javadoc and relevant documentation. So now I hope it makes more sense why I pointed you to the JIRA river 🙂 Regards, Lukas [1] [https://github.com/jbossorg/elasticsearch-river-jira](https://github.com/jbossorg/elasticsearch-river-jira) [2] [https://github.com/jbossorg/elasticsearch-river-remote](https://github.com/jbossorg/elasticsearch-river-remote) [3] [https://github.com/jbossorg/elasticsearch-river-jira#management-rest-api](https://github.com/jbossorg/elasticsearch-river-jira#management-rest-api) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [September 24, 2013, 10:18pm UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/10 "2013-09-24T22:18:27Z")
Hi Lukáš, I did not dig deep enough into the code. I now see that the relevant classes are inside the subfolders of mgm. Since there are many classes to implement, once I did not see the explosion of classes in jira or mgm I assumed it was just a standard river. My implementation has been working smoothly, but I will still look at the code to see if there is anything else that can be learned. It did take quite some work to figure out since there is no documentation. 🙂 Have not seen any other plugins that work at the node level, at least not at a glance (such as the JIRA river). I want to implement some of the other types of TransportActions just to see how they behave and how they differ from each other. For example, what is the difference between TransportSingleCustomOperationAction and TransportBroadcastOperationAction? Cheers, Ivan On Tue, Sep 24, 2013 at 1:37 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Hi Ivan, > > yes I agree, but let me explain why I pointed out the JIRA river. > > Rivers are relatively easy that is true - but only as long as you are fine > with its basic contract, i.e. start() and close() operations. Once you want > more fine grained control of your river then it gets more complicated. In > our rivers: JIRA river [1] and Remote river [2] (both are basically > similar, the former is just more specific to JIRA as a data source) we also > wanted to provide REST API for management of individual river instances and > reporting status [3]. Extending REST API is again simple - by registering > new actions on RestModule. But the question is what should happen when the > REST request is received on the node which is not running the river > singleton. Now, from what I have seen there are basically two approaches > how ppl do go about this. > > The first one is to store client action as a document into cluster (for > example into \_river index) and have the river check for such documents in > periodic intervals. The river uses the search API to learn if there was any > request to execute some specific action. > > The second way (which we used) is a bit different. In nutshell it includes > full propagation from RestRequest to ClusterAction (and this includes > NodeOperationRequest/Response serialization as well). I think this provides > better options in terms of river control and reporting the status from > river. > > May be there are better ways of doing this but as of now we do not know > about them. Also I am not aware of any other plugin that would do it in > similar way. It took Vlastimil quite some work to get there, one of the > biggest issues was missing Javadoc and relevant documentation. > > So now I hope it makes more sense why I pointed you to the JIRA river 🙂 > > Regards, > Lukas > > [1] [GitHub - searchisko/elasticsearch-river-jira: JIRA River Plugin for Elasticsearch](https://github.com/jbossorg/elasticsearch-river-jira) > [2] [GitHub - searchisko/elasticsearch-river-remote: Universal remote system indexing River Plugin for Elasticsearch](https://github.com/jbossorg/elasticsearch-river-remote) > [3] > [GitHub - searchisko/elasticsearch-river-jira: JIRA River Plugin for Elasticsearch](https://github.com/jbossorg/elasticsearch-river-jira#management-rest-api) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:14am UTC](https://discuss.elastic.co/t/node-plugin-not-propagating-requests/13681/11 "2017-07-06T02:14:50Z")