Since rivers are just a style of execution of ES plugins, I am sure there
will be replacements at the plugin level, after rivers are officially
deprecated.
Logstash or custom middleware, or the JDBC feed mode, are push-style
applications, which can ingest data into ES, without the requirement of a
plugin. I do not think that push-style is always feasible to replace river
style execution. An example is the multi tenancy scenario, where a provider
wants a single piece of SaaS code to maintain for all tenants. Or stateful
indexing, where the last known state of the indexing should be available at
the ES side. You could implement a pull-style framework within the ES JVM,
with provisioning, job processing, drivers, dispatchers, multi tenancy etc.
(prominent example would be a web harvester). In this scenario, no change
at middleware would be required, or adding extra machinery to the indexing
chain, like Logstash, or JDBC feed mode, because all is hosted within ES
JVM. So in the end it is all a matter of style how to index docs. Both push
or pull approaches do not interfere.
Jörg
On Sun, Aug 31, 2014 at 6:10 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:
Nitin,
If I am not mistaken rivers in ES will be deprecated. So in any case river
is not a good way to go for design of future application. We are using
rivers quite a lot in our project and we will be moving the functionality
to middleware layer.
Regards,
Lukáš
Dne 31.8.2014 3:58 "Nitin Maheshwari" ask4nitin@gmail.com napsal(a):
Thanks for the reference, its helpful.
On Thu, Aug 28, 2014 at 12:33 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:
I work on such a component which can handle jobs and delegate them to
other nodes but no working implementation available yet.
For a service component that is able to maintain state in the cluster
state, see RiverState class in JDBC plugin.
Jörg
On Thu, Aug 28, 2014 at 8:08 AM, Nitin Maheshwari ask4nitin@gmail.com
wrote:
Thanks Jörg for your quick and timely response.
I am new to ES, can you point me to any reference implementation for
the ES service component?
Once again thanks for the help.
Nitin
On Tuesday, 26 August 2014 15:07:05 UTC+5:30, Jörg Prante wrote:
For multi tenant, the river concept is awkward. River is a singleton
and is bound to single user execution, and you are right, creating river
instances per DB and per index does not scale.
There are several options:
-
write a more sophisticated plugin which acts as a service and not as
a singleton. The ES service component, which would maintain state in the
cluster state, could accept job requests where each job request is
equivalent to a JDBC pull. The job requests are delegated to a node which
is not very busy with jobs (load balancing). The code of the JDBC river can
be reused for that.
-
write a separate middleware for your tenants where they can have
separate access to the DB and prepare ES JSON bulk files from (maybe be by
REST API calls similar in style to ES). This would be a domain specific
solution but offers most flexibility to the tenants, they are free to
decide how and when to create and index the data from DB.
Jörg
On Tue, Aug 26, 2014 at 11:21 AM, Nitin Maheshwari ask4...@gmail.com
wrote:
Hi Jörg,
I am working on a multi tenant application where each tenant has its
own database. I am planning to use ES for indexing the data, and JDBC river
for doing periodic bulk indexing. I do not want to create one river per DB
per object type. This will lead to too many rivers.
I wanted to modify the JDBC river so that I can give parent DB
location, where all tenant db connection information is available. And then
inside the river, modify it such that a feader thread is created for each
river.
Do you see any issue with this or do you have any other
recommendation?
Thanks,
Nitin
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/w88yWcbx4lM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Nitin (Nits)
http://nitinmaheshwari.in
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHxPCeYiMj4wznCaLyJrzJ%3DcHwpzfN1C5_%2Bs_wnmy1RxA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.