# Plugin broadcasting/scalability **URL:** \ **Category:** Elasticsearch\ **Created:** [December 23, 2014, 9:23am UTC](https://discuss.elastic.co/t/plugin-broadcasting-scalability/21364 "2014-12-23T09:23:57Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![aken](https://avatars.discourse-cdn.com/v4/letter/a/b38774/32.png) [@aken](https://discuss.elastic.co/u/aken)\ **Post date:** [December 23, 2014, 9:23am UTC](https://discuss.elastic.co/t/plugin-broadcasting-scalability/21364/1 "2014-12-23T09:23:57Z")
A while ago I used a plugin on a project to control the migration to, and the maintenance of, a secondary data source/index in ElasticSearch. It worked really well and avoided lots of messy dependency management in the host application, as all the logic was "hidden" behind an ES rest end-point. But I was never really sure how scalable this was. I have two questions: 1.) when I register a plugin, is that available for all nodes? (I'm assuming "yes", which would mean controlling parallel/overlapping call outs is important) 2.) when i call my end-point, I have a Client Object passed in the constructor: when I debug this, it is an instance of NodeClient which presumably means I am working on a single node. Is it possible to construct a TransportClient from this, so that I can address more than node and take advantage of e.g. bulk imports in parallel? Regarding 2.), I've had a look at the JDBC River Code and the Feeder mode (addressing the cluster from a component running in a separate JVM) seems to be there precisely because of this drawback. The River mode seems to work off one node like my plugin did/does. Is my understanding correct? Andrew -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 23, 2014, 1:38pm UTC](https://discuss.elastic.co/t/plugin-broadcasting-scalability/21364/2 "2014-12-23T13:38:25Z")
1. Plugins only run on the node they are installed on. For this, a plugin should be installed on all nodes, no matter if it is used on all nodes or not. Plugins can have actions implemented that can be addressed by other nodes (broadcast operations). 2. The NodeClient you get is sitting on the local node but can potentially address all nodes. Bulk imports automatically use nodes in parallel, i.e. they forward actions to the nodes where the shards of the addressed index of an operation live. The JDBC feeder mode works off the cluster and uses a TransportClient. The reason for the feeder mode is that it works without being dependent on an ES node life cycle - rivers are flaky when they are forced to hop to another node and are restarted when the river nodes goes down. Rivers use NodeClient instances and the bulk indexing scales well. The part that does not scale in rivers is the river management by ES and the "pull" style, the non-existent distributed architecture of the fetch phase. So, a river instance soon becomes a bottleneck or a single point of failure, a design which does not fit to the well-designed architecture of the rest of the ES system. Jörg On Tue, Dec 23, 2014 at 10:23 AM, AndrewK [kenworthyas@gmail.com](mailto:kenworthyas@gmail.com) wrote: > A while ago I used a plugin on a project to control the migration to, and > the maintenance of, a secondary data source/index in Elasticsearch. It > worked really well and avoided lots of messy dependency management in the > host application, as all the logic was "hidden" behind an ES rest end-point. > > But I was never really sure how scalable this was. I have two questions: > > 1.) when I register a plugin, is that available for all nodes? (I'm > assuming "yes", which would mean controlling parallel/overlapping call outs > is important) > > 2.) when i call my end-point, I have a Client Object passed in the > constructor: when I debug this, it is an instance of NodeClient which > presumably means I am working on a single node. Is it possible to construct > a TransportClient from this, so that I can address more than node and take > advantage of e.g. bulk imports in parallel? > > Regarding 2.), I've had a look at the JDBC River Code and the Feeder mode > (addressing the cluster from a component running in a separate JVM) seems > to be there precisely because of this drawback. The River mode seems to > work off one node like my plugin did/does. > > Is my understanding correct? > > Andrew > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHnkAd6UoFFj%2Biv%2B5hZo63X%3D9\_LiwTM3sQiE-OFbZPGNw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHnkAd6UoFFj%2Biv%2B5hZo63X%3D9_LiwTM3sQiE-OFbZPGNw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![aken](https://avatars.discourse-cdn.com/v4/letter/a/b38774/32.png) [@aken](https://discuss.elastic.co/u/aken)\ **Post date:** [December 23, 2014, 3:11pm UTC](https://discuss.elastic.co/t/plugin-broadcasting-scalability/21364/3 "2014-12-23T15:11:47Z")
Thank you for the information - that has clarified things for me! Andrew Am Dienstag, 23. Dezember 2014 14:38:34 UTC+1 schrieb Jörg Prante: > 1. Plugins only run on the node they are installed on. For this, a plugin > should be installed on all nodes, no matter if it is used on all nodes or > not. Plugins can have actions implemented that can be addressed by other > nodes (broadcast operations). > > 2. The NodeClient you get is sitting on the local node but can potentially > address all nodes. Bulk imports automatically use nodes in parallel, i.e. > they forward actions to the nodes where the shards of the addressed index > of an operation live. > > The JDBC feeder mode works off the cluster and uses a TransportClient. The > reason for the feeder mode is that it works without being dependent on an > ES node life cycle - rivers are flaky when they are forced to hop to > another node and are restarted when the river nodes goes down. > > Rivers use NodeClient instances and the bulk indexing scales well. The > part that does not scale in rivers is the river management by ES and the > "pull" style, the non-existent distributed architecture of the fetch phase. > So, a river instance soon becomes a bottleneck or a single point of > failure, a design which does not fit to the well-designed architecture of > the rest of the ES system. > > Jörg > > On Tue, Dec 23, 2014 at 10:23 AM, AndrewK \<[kenwo...@gmail.com](mailto:kenwo...@gmail.com) > \\> wrote: > > > A while ago I used a plugin on a project to control the migration to, and > > the maintenance of, a secondary data source/index in Elasticsearch. It > > worked really well and avoided lots of messy dependency management in the > > host application, as all the logic was "hidden" behind an ES rest end-point. > > > > But I was never really sure how scalable this was. I have two questions: > > > > 1.) when I register a plugin, is that available for all nodes? (I'm > > assuming "yes", which would mean controlling parallel/overlapping call outs > > is important) > > > > 2.) when i call my end-point, I have a Client Object passed in the > > constructor: when I debug this, it is an instance of NodeClient which > > presumably means I am working on a single node. Is it possible to construct > > a TransportClient from this, so that I can address more than node and take > > advantage of e.g. bulk imports in parallel? > > > > Regarding 2.), I've had a look at the JDBC River Code and the Feeder mode > > (addressing the cluster from a component running in a separate JVM) seems > > to be there precisely because of this drawback. The River mode seems to > > work off one node like my plugin did/does. > > > > Is my understanding correct? > > > > Andrew > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aee5743a-f706-4034-b139-4a0582d6223a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aee5743a-f706-4034-b139-4a0582d6223a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:42am UTC](https://discuss.elastic.co/t/plugin-broadcasting-scalability/21364/4 "2017-07-06T00:42:12Z")