# Multi Tenant DB and JDBC River **URL:** \ **Category:** Elasticsearch\ **Created:** [August 26, 2014, 9:21am UTC](https://discuss.elastic.co/t/multi-tenant-db-and-jdbc-river/19456 "2014-08-26T09:21:00Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Nitin\_Maheshwari](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@Nitin\_Maheshwari](https://discuss.elastic.co/u/Nitin_Maheshwari)\ **Post date:** [August 26, 2014, 9:21am UTC](https://discuss.elastic.co/t/multi-tenant-db-and-jdbc-river/19456/1 "2014-08-26T09:21:00Z")
Hi Jörg, I am working on a multi tenant application where each tenant has its own database. I am planning to use ES for indexing the data, and JDBC river for doing periodic bulk indexing. I do not want to create one river per DB per object type. This will lead to too many rivers. I wanted to modify the JDBC river so that I can give parent DB location, where all tenant db connection information is available. And then inside the river, modify it such that a feader thread is created for each river. Do you see any issue with this or do you have any other recommendation? Thanks, Nitin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 26, 2014, 9:36am UTC](https://discuss.elastic.co/t/multi-tenant-db-and-jdbc-river/19456/2 "2014-08-26T09:36:57Z")
For multi tenant, the river concept is awkward. River is a singleton and is bound to single user execution, and you are right, creating river instances per DB and per index does not scale. There are several options: - write a more sophisticated plugin which acts as a service and not as a singleton. The ES service component, which would maintain state in the cluster state, could accept job requests where each job request is equivalent to a JDBC pull. The job requests are delegated to a node which is not very busy with jobs (load balancing). The code of the JDBC river can be reused for that. - write a separate middleware for your tenants where they can have separate access to the DB and prepare ES JSON bulk files from (maybe be by REST API calls similar in style to ES). This would be a domain specific solution but offers most flexibility to the tenants, they are free to decide how and when to create and index the data from DB. Jörg On Tue, Aug 26, 2014 at 11:21 AM, Nitin Maheshwari [ask4nitin@gmail.com](mailto:ask4nitin@gmail.com) wrote: > Hi Jörg, > > I am working on a multi tenant application where each tenant has its own > database. I am planning to use ES for indexing the data, and JDBC river for > doing periodic bulk indexing. I do not want to create one river per DB per > object type. This will lead to too many rivers. > > I wanted to modify the JDBC river so that I can give parent DB location, > where all tenant db connection information is available. And then inside > the river, modify it such that a feader thread is created for each river. > > Do you see any issue with this or do you have any other recommendation? > > Thanks, > Nitin > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEZumpR1oazTuVn6Ofad71jgEMkSBOKARizK9-gOFpVsA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEZumpR1oazTuVn6Ofad71jgEMkSBOKARizK9-gOFpVsA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Nitin\_Maheshwari](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@Nitin\_Maheshwari](https://discuss.elastic.co/u/Nitin_Maheshwari)\ **Post date:** [August 28, 2014, 6:08am UTC](https://discuss.elastic.co/t/multi-tenant-db-and-jdbc-river/19456/3 "2014-08-28T06:08:18Z")
Thanks Jörg for your quick and timely response. I am new to ES, can you point me to any reference implementation for the ES service component? Once again thanks for the help. Nitin On Tuesday, 26 August 2014 15:07:05 UTC+5:30, Jörg Prante wrote: > For multi tenant, the river concept is awkward. River is a singleton and > is bound to single user execution, and you are right, creating river > instances per DB and per index does not scale. > > There are several options: > > - write a more sophisticated plugin which acts as a service and not as a > singleton. The ES service component, which would maintain state in the > cluster state, could accept job requests where each job request is > equivalent to a JDBC pull. The job requests are delegated to a node which > is not very busy with jobs (load balancing). The code of the JDBC river can > be reused for that. > > - write a separate middleware for your tenants where they can have > separate access to the DB and prepare ES JSON bulk files from (maybe be by > REST API calls similar in style to ES). This would be a domain specific > solution but offers most flexibility to the tenants, they are free to > decide how and when to create and index the data from DB. > > Jörg > > On Tue, Aug 26, 2014 at 11:21 AM, Nitin Maheshwari \<[ask4...@gmail.com](mailto:ask4...@gmail.com) > \\> wrote: > > > Hi Jörg, > > > > I am working on a multi tenant application where each tenant has its own > > database. I am planning to use ES for indexing the data, and JDBC river for > > doing periodic bulk indexing. I do not want to create one river per DB per > > object type. This will lead to too many rivers. > > > > I wanted to modify the JDBC river so that I can give parent DB location, > > where all tenant db connection information is available. And then inside > > the river, modify it such that a feader thread is created for each river. > > > > Do you see any issue with this or do you have any other recommendation? > > > > Thanks, > > Nitin > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 28, 2014, 7:03am UTC](https://discuss.elastic.co/t/multi-tenant-db-and-jdbc-river/19456/4 "2014-08-28T07:03:37Z")
I work on such a component which can handle jobs and delegate them to other nodes but no working implementation available yet. For a service component that is able to maintain state in the cluster state, see RiverState class in JDBC plugin. Jörg On Thu, Aug 28, 2014 at 8:08 AM, Nitin Maheshwari [ask4nitin@gmail.com](mailto:ask4nitin@gmail.com) wrote: > Thanks Jörg for your quick and timely response. > > I am new to ES, can you point me to any reference implementation for the > ES service component? > > Once again thanks for the help. > > Nitin > > On Tuesday, 26 August 2014 15:07:05 UTC+5:30, Jörg Prante wrote: > > > For multi tenant, the river concept is awkward. River is a singleton and > > is bound to single user execution, and you are right, creating river > > instances per DB and per index does not scale. > > > > There are several options: > > > > - write a more sophisticated plugin which acts as a service and not as a > > singleton. The ES service component, which would maintain state in the > > cluster state, could accept job requests where each job request is > > equivalent to a JDBC pull. The job requests are delegated to a node which > > is not very busy with jobs (load balancing). The code of the JDBC river can > > be reused for that. > > > > - write a separate middleware for your tenants where they can have > > separate access to the DB and prepare ES JSON bulk files from (maybe be by > > REST API calls similar in style to ES). This would be a domain specific > > solution but offers most flexibility to the tenants, they are free to > > decide how and when to create and index the data from DB. > > > > Jörg > > > > On Tue, Aug 26, 2014 at 11:21 AM, Nitin Maheshwari [ask4...@gmail.com](mailto:ask4...@gmail.com) > > wrote: > > > > > Hi Jörg, > > > > > > I am working on a multi tenant application where each tenant has its own > > > database. I am planning to use ES for indexing the data, and JDBC river for > > > doing periodic bulk indexing. I do not want to create one river per DB per > > > object type. This will lead to too many rivers. > > > > > > I wanted to modify the JDBC river so that I can give parent DB location, > > > where all tenant db connection information is available. And then inside > > > the river, modify it such that a feader thread is created for each river. > > > > > > Do you see any issue with this or do you have any other recommendation? > > > > > > Thanks, > > > Nitin > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO\_TJNthcvD1RK-pHWruwL0gx\_Rew%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Nitin\_Maheshwari](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@Nitin\_Maheshwari](https://discuss.elastic.co/u/Nitin_Maheshwari)\ **Post date:** [August 31, 2014, 1:58am UTC](https://discuss.elastic.co/t/multi-tenant-db-and-jdbc-river/19456/5 "2014-08-31T01:58:05Z")
Thanks for the reference, its helpful. On Thu, Aug 28, 2014 at 12:33 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > I work on such a component which can handle jobs and delegate them to > other nodes but no working implementation available yet. > > For a service component that is able to maintain state in the cluster > state, see RiverState class in JDBC plugin. > > Jörg > > On Thu, Aug 28, 2014 at 8:08 AM, Nitin Maheshwari [ask4nitin@gmail.com](mailto:ask4nitin@gmail.com) > wrote: > > > Thanks Jörg for your quick and timely response. > > > > I am new to ES, can you point me to any reference implementation for the > > ES service component? > > > > Once again thanks for the help. > > > > Nitin > > > > On Tuesday, 26 August 2014 15:07:05 UTC+5:30, Jörg Prante wrote: > > > > > For multi tenant, the river concept is awkward. River is a singleton and > > > is bound to single user execution, and you are right, creating river > > > instances per DB and per index does not scale. > > > > > > There are several options: > > > > > > - write a more sophisticated plugin which acts as a service and not as a > > > singleton. The ES service component, which would maintain state in the > > > cluster state, could accept job requests where each job request is > > > equivalent to a JDBC pull. The job requests are delegated to a node which > > > is not very busy with jobs (load balancing). The code of the JDBC river can > > > be reused for that. > > > > > > - write a separate middleware for your tenants where they can have > > > separate access to the DB and prepare ES JSON bulk files from (maybe be by > > > REST API calls similar in style to ES). This would be a domain specific > > > solution but offers most flexibility to the tenants, they are free to > > > decide how and when to create and index the data from DB. > > > > > > Jörg > > > > > > On Tue, Aug 26, 2014 at 11:21 AM, Nitin Maheshwari [ask4...@gmail.com](mailto:ask4...@gmail.com) > > > wrote: > > > > > > > Hi Jörg, > > > > > > > > I am working on a multi tenant application where each tenant has its > > > > own database. I am planning to use ES for indexing the data, and JDBC river > > > > for doing periodic bulk indexing. I do not want to create one river per DB > > > > per object type. This will lead to too many rivers. > > > > > > > > I wanted to modify the JDBC river so that I can give parent DB > > > > location, where all tenant db connection information is available. And then > > > > inside the river, modify it such that a feader thread is created for each > > > > river. > > > > > > > > Do you see any issue with this or do you have any other recommendation? > > > > > > > > Thanks, > > > > Nitin > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/w88yWcbx4lM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/w88yWcbx4lM/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO\_TJNthcvD1RK-pHWruwL0gx\_Rew%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO\_TJNthcvD1RK-pHWruwL0gx\_Rew%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Nitin (Nits) [http://nitinmaheshwari.in](http://nitinmaheshwari.in) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [August 31, 2014, 4:10am UTC](https://discuss.elastic.co/t/multi-tenant-db-and-jdbc-river/19456/6 "2014-08-31T04:10:28Z")
Nitin, If I am not mistaken rivers in ES will be deprecated. So in any case river is not a good way to go for design of future application. We are using rivers quite a lot in our project and we will be moving the functionality to middleware layer. Regards, Lukáš Dne 31.8.2014 3:58 "Nitin Maheshwari" [ask4nitin@gmail.com](mailto:ask4nitin@gmail.com) napsal(a): > Thanks for the reference, its helpful. > > On Thu, Aug 28, 2014 at 12:33 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > I work on such a component which can handle jobs and delegate them to > > other nodes but no working implementation available yet. > > > > For a service component that is able to maintain state in the cluster > > state, see RiverState class in JDBC plugin. > > > > Jörg > > > > On Thu, Aug 28, 2014 at 8:08 AM, Nitin Maheshwari [ask4nitin@gmail.com](mailto:ask4nitin@gmail.com) > > wrote: > > > > > Thanks Jörg for your quick and timely response. > > > > > > I am new to ES, can you point me to any reference implementation for the > > > ES service component? > > > > > > Once again thanks for the help. > > > > > > Nitin > > > > > > On Tuesday, 26 August 2014 15:07:05 UTC+5:30, Jörg Prante wrote: > > > > > > > For multi tenant, the river concept is awkward. River is a singleton > > > > and is bound to single user execution, and you are right, creating river > > > > instances per DB and per index does not scale. > > > > > > > > There are several options: > > > > > > > > - write a more sophisticated plugin which acts as a service and not as > > > > a singleton. The ES service component, which would maintain state in the > > > > cluster state, could accept job requests where each job request is > > > > equivalent to a JDBC pull. The job requests are delegated to a node which > > > > is not very busy with jobs (load balancing). The code of the JDBC river can > > > > be reused for that. > > > > > > > > - write a separate middleware for your tenants where they can have > > > > separate access to the DB and prepare ES JSON bulk files from (maybe be by > > > > REST API calls similar in style to ES). This would be a domain specific > > > > solution but offers most flexibility to the tenants, they are free to > > > > decide how and when to create and index the data from DB. > > > > > > > > Jörg > > > > > > > > On Tue, Aug 26, 2014 at 11:21 AM, Nitin Maheshwari [ask4...@gmail.com](mailto:ask4...@gmail.com) > > > > wrote: > > > > > > > > > Hi Jörg, > > > > > > > > > > I am working on a multi tenant application where each tenant has its > > > > > own database. I am planning to use ES for indexing the data, and JDBC river > > > > > for doing periodic bulk indexing. I do not want to create one river per DB > > > > > per object type. This will lead to too many rivers. > > > > > > > > > > I wanted to modify the JDBC river so that I can give parent DB > > > > > location, where all tenant db connection information is available. And then > > > > > inside the river, modify it such that a feader thread is created for each > > > > > river. > > > > > > > > > > Do you see any issue with this or do you have any other recommendation? > > > > > > > > > > Thanks, > > > > > Nitin > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/w88yWcbx4lM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/w88yWcbx4lM/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO\_TJNthcvD1RK-pHWruwL0gx\_Rew%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO\_TJNthcvD1RK-pHWruwL0gx\_Rew%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Nitin (Nits) > [http://nitinmaheshwari.in](http://nitinmaheshwari.in) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 31, 2014, 10:02am UTC](https://discuss.elastic.co/t/multi-tenant-db-and-jdbc-river/19456/7 "2014-08-31T10:02:16Z")
Since rivers are just a style of execution of ES plugins, I am sure there will be replacements at the plugin level, after rivers are officially deprecated. Logstash or custom middleware, or the JDBC feed mode, are push-style applications, which can ingest data into ES, without the requirement of a plugin. I do not think that push-style is always feasible to replace river style execution. An example is the multi tenancy scenario, where a provider wants a single piece of SaaS code to maintain for all tenants. Or stateful indexing, where the last known state of the indexing should be available at the ES side. You could implement a pull-style framework within the ES JVM, with provisioning, job processing, drivers, dispatchers, multi tenancy etc. (prominent example would be a web harvester). In this scenario, no change at middleware would be required, or adding extra machinery to the indexing chain, like Logstash, or JDBC feed mode, because all is hosted within ES JVM. So in the end it is all a matter of style how to index docs. Both push or pull approaches do not interfere. Jörg On Sun, Aug 31, 2014 at 6:10 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Nitin, > > If I am not mistaken rivers in ES will be deprecated. So in any case river > is not a good way to go for design of future application. We are using > rivers quite a lot in our project and we will be moving the functionality > to middleware layer. > > Regards, > Lukáš > Dne 31.8.2014 3:58 "Nitin Maheshwari" [ask4nitin@gmail.com](mailto:ask4nitin@gmail.com) napsal(a): > > Thanks for the reference, its helpful. > > > On Thu, Aug 28, 2014 at 12:33 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > I work on such a component which can handle jobs and delegate them to > > > other nodes but no working implementation available yet. > > > > > > For a service component that is able to maintain state in the cluster > > > state, see RiverState class in JDBC plugin. > > > > > > Jörg > > > > > > On Thu, Aug 28, 2014 at 8:08 AM, Nitin Maheshwari [ask4nitin@gmail.com](mailto:ask4nitin@gmail.com) > > > wrote: > > > > > > > Thanks Jörg for your quick and timely response. > > > > > > > > I am new to ES, can you point me to any reference implementation for > > > > the ES service component? > > > > > > > > Once again thanks for the help. > > > > > > > > Nitin > > > > > > > > On Tuesday, 26 August 2014 15:07:05 UTC+5:30, Jörg Prante wrote: > > > > > > > > > For multi tenant, the river concept is awkward. River is a singleton > > > > > and is bound to single user execution, and you are right, creating river > > > > > instances per DB and per index does not scale. > > > > > > > > > > There are several options: > > > > > > > > > > - write a more sophisticated plugin which acts as a service and not as > > > > > a singleton. The ES service component, which would maintain state in the > > > > > cluster state, could accept job requests where each job request is > > > > > equivalent to a JDBC pull. The job requests are delegated to a node which > > > > > is not very busy with jobs (load balancing). The code of the JDBC river can > > > > > be reused for that. > > > > > > > > > > - write a separate middleware for your tenants where they can have > > > > > separate access to the DB and prepare ES JSON bulk files from (maybe be by > > > > > REST API calls similar in style to ES). This would be a domain specific > > > > > solution but offers most flexibility to the tenants, they are free to > > > > > decide how and when to create and index the data from DB. > > > > > > > > > > Jörg > > > > > > > > > > On Tue, Aug 26, 2014 at 11:21 AM, Nitin Maheshwari [ask4...@gmail.com](mailto:ask4...@gmail.com) > > > > > wrote: > > > > > > > > > > > Hi Jörg, > > > > > > > > > > > > I am working on a multi tenant application where each tenant has its > > > > > > own database. I am planning to use ES for indexing the data, and JDBC river > > > > > > for doing periodic bulk indexing. I do not want to create one river per DB > > > > > > per object type. This will lead to too many rivers. > > > > > > > > > > > > I wanted to modify the JDBC river so that I can give parent DB > > > > > > location, where all tenant db connection information is available. And then > > > > > > inside the river, modify it such that a feader thread is created for each > > > > > > river. > > > > > > > > > > > > Do you see any issue with this or do you have any other > > > > > > recommendation? > > > > > > > > > > > > Thanks, > > > > > > Nitin > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473% > > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/w88yWcbx4lM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/w88yWcbx4lM/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO\_TJNthcvD1RK-pHWruwL0gx\_Rew%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO\_TJNthcvD1RK-pHWruwL0gx\_Rew%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Nitin (Nits) > > [http://nitinmaheshwari.in](http://nitinmaheshwari.in) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHxPCeYiMj4wznCaLyJrzJ%3DcHwpzfN1C5\_%2Bs\_wnmy1RxA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHxPCeYiMj4wznCaLyJrzJ%3DcHwpzfN1C5_%2Bs_wnmy1RxA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:05am UTC](https://discuss.elastic.co/t/multi-tenant-db-and-jdbc-river/19456/8 "2017-07-06T01:05:18Z")