# Index documents out of postgresql database but without a river plugin **URL:** \ **Category:** Elasticsearch\ **Created:** [November 14, 2013, 12:27pm UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405 "2013-11-14T12:27:47Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Isabella](https://avatars.discourse-cdn.com/v4/letter/i/ecb155/32.png) [@Isabella](https://discuss.elastic.co/u/Isabella)\ **Post date:** [November 14, 2013, 12:27pm UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/1 "2013-11-14T12:27:47Z")
Hello! I have to make an plugin which indexes data from my postgresql database, but it should not be a River Plugin. My problem is, that I don't know how to let my plugin know, that I have created a mapping, that defines my jdbc driver, url, password etc. and how to get those properties to connect to my database in my plugin java code? Does anyone know if I can define those jdbc properties in a mapping and read it out or do I have to do it in another way? I really would appreciate your help! Thanks a lot. Regards Isabella -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 14, 2013, 2:56pm UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/2 "2013-11-14T14:56:16Z")
Question is "why a plugin?" It does not bring so much value to elasticsearch internals. I would start with a simple Main() class (let's say a batch for example). -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 14 novembre 2013 at 13:27:52, Isabella ([isi.huber90@gmail.com](mailto:isi.huber90@gmail.com)) a écrit: Hello! I have to make an plugin which indexes data from my postgresql database, but it should not be a River Plugin. My problem is, that I don't know how to let my plugin know, that I have created a mapping, that defines my jdbc driver, url, password etc. and how to get those properties to connect to my database in my plugin java code? Does anyone know if I can define those jdbc properties in a mapping and read it out or do I have to do it in another way? I really would appreciate your help! Thanks a lot. ## Regards Isabella You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 14, 2013, 6:28pm UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/3 "2013-11-14T18:28:06Z")
Probably you want to store the JDBC settings in the cluster state? That is exactly what a river plugin does 🙂 Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [November 14, 2013, 8:16pm UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/4 "2013-11-14T20:16:25Z")
We like to call this a "shoveller" (or shoveler, depends on your locale), as in "shovelling data into ES" On Thu, Nov 14, 2013 at 4:56 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > Question is "why a plugin?" > > It does not bring so much value to elasticsearch internals. > > I would start with a simple Main() class (let's say a batch for example). > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > Le 14 novembre 2013 at 13:27:52, Isabella ([isi.huber90@gmail.com](mailto:isi.huber90@gmail.com)//isi.huber90@gmail.com) > a écrit: > > Hello! > > I have to make an plugin which indexes data from my postgresql database, > but it should not be a River Plugin. My problem is, that I don't know how > to let my plugin know, that I have created a mapping, that defines my jdbc > driver, url, password etc. and how to get those properties to connect to my > database in my plugin java code? > > Does anyone know if I can define those jdbc properties in a mapping and > read it out or do I have to do it in another way? > > I really would appreciate your help! > Thanks a lot. > > ## Regards Isabella > > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Isabella](https://avatars.discourse-cdn.com/v4/letter/i/ecb155/32.png) [@Isabella](https://discuss.elastic.co/u/Isabella)\ **Post date:** [November 18, 2013, 10:11am UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/5 "2013-11-18T10:11:42Z")
Thanks for your answers. Unfortunately I don't know, why it has to be a normal plugin instead of a river plugin. I only got the instructions, that it has to be a normal plugin which synchronizes data from postgresql. I have to implement a normal plugin, that should read the properties to connect with the database out of a config file. For instance, how it is done in the hadoop plugin. > **[GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and...](https://github.com/elastic/elasticsearch-hadoop)** > > :elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop - GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated wi... Is there a possibility to configure the database connection properties in such a config file? My second problem is, that I don't know which module I have to use for my plugin, so that my plugin actually does something when I start ES. Is there any list of available modules for ES? I only know a few modules like, RiverModule, AnalysisModule, ScriptModule and RestModule, but I don't think that these modules are the right for my plugin. Another problem is how can I address my plugin? Do I need a mapping to say ES that my plugin should do something? I am sorry about my stupid and mabye a litte bit confusing questions. But my problem is, that I don't know how to initialize my plugin. If it would be a river plugin, I have to define a mapping with a type, that is also defined in the java code, when the riverModule is registered, am I right? But if I have a normal plugin I don't know which module I have to take or which modules are available. I really would appreciate any help. Thanks a lot. Best Regards, Isabella Am Donnerstag, 14. November 2013 21:16:25 UTC+1 schrieb Itamar Syn-Hershko: > We like to call this a "shoveller" (or shoveler, depends on your locale), > as in "shovelling data into ES" > > On Thu, Nov 14, 2013 at 4:56 PM, David Pilato \<[da...@pilato.fr](mailto:da...@pilato.fr)\ > > > wrote: > > > Question is "why a plugin?" > > > > It does not bring so much value to elasticsearch internals. > > > > I would start with a simple Main() class (let's say a batch for example). > > > > -- > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > > > Le 14 novembre 2013 at 13:27:52, Isabella ([isi.h...@gmail.com](mailto:isi.h...@gmail.com)\) > > a écrit: > > > > Hello! > > > > I have to make an plugin which indexes data from my postgresql database, > > but it should not be a River Plugin. My problem is, that I don't know how > > to let my plugin know, that I have created a mapping, that defines my jdbc > > driver, url, password etc. and how to get those properties to connect to my > > database in my plugin java code? > > > > Does anyone know if I can define those jdbc properties in a mapping and > > read it out or do I have to do it in another way? > > > > I really would appreciate your help! > > Thanks a lot. > > > > ## Regards Isabella > > > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 18, 2013, 11:10am UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/6 "2013-11-18T11:10:24Z")
Hey Isabella, I suppose that you are a contractor for a company and that your contract is to build a plugin, right? A river plugin is a plugin. So you respect the contract. That said, I would really recommend that you give advices to your client that it could really be better to have that logic outside elasticsearch itself. My 2 cents. -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 18 novembre 2013 at 11:11:45, Isabella ([isi.huber90@gmail.com](mailto:isi.huber90@gmail.com)) a écrit: Thanks for your answers. Unfortunately I don't know, why it has to be a normal plugin instead of a river plugin. I only got the instructions, that it has to be a normal plugin which synchronizes data from postgresql. I have to implement a normal plugin, that should read the properties to connect with the database out of a config file. For instance, how it is done in the hadoop plugin. [https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop) Is there a possibility to configure the database connection properties in such a config file? My second problem is, that I don't know which module I have to use for my plugin, so that my plugin actually does something when I start ES. Is there any list of available modules for ES? I only know a few modules like, RiverModule, AnalysisModule, ScriptModule and RestModule, but I don't think that these modules are the right for my plugin. Another problem is how can I address my plugin? Do I need a mapping to say ES that my plugin should do something? I am sorry about my stupid and mabye a litte bit confusing questions. But my problem is, that I don't know how to initialize my plugin. If it would be a river plugin, I have to define a mapping with a type, that is also defined in the java code, when the riverModule is registered, am I right? But if I have a normal plugin I don't know which module I have to take or which modules are available. I really would appreciate any help. Thanks a lot. Best Regards, Isabella Am Donnerstag, 14. November 2013 21:16:25 UTC+1 schrieb Itamar Syn-Hershko: We like to call this a "shoveller" (or shoveler, depends on your locale), as in "shovelling data into ES" On Thu, Nov 14, 2013 at 4:56 PM, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: Question is "why a plugin?" It does not bring so much value to elasticsearch internals. I would start with a simple Main() class (let's say a batch for example). -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 14 novembre 2013 at 13:27:52, Isabella ([isi.h...@gmail.com](mailto:isi.h...@gmail.com)) a écrit: Hello! I have to make an plugin which indexes data from my postgresql database, but it should not be a River Plugin. My problem is, that I don't know how to let my plugin know, that I have created a mapping, that defines my jdbc driver, url, password etc. and how to get those properties to connect to my database in my plugin java code? Does anyone know if I can define those jdbc properties in a mapping and read it out or do I have to do it in another way? I really would appreciate your help! Thanks a lot. ## Regards Isabella ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Isabella](https://avatars.discourse-cdn.com/v4/letter/i/ecb155/32.png) [@Isabella](https://discuss.elastic.co/u/Isabella)\ **Post date:** [November 18, 2013, 11:28am UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/7 "2013-11-18T11:28:20Z")
Hey David! Thanks a lot for your answer and yes your are right with that. Ok I will do that. Thank you. Isabella 2013/11/18 David Pilato [david@pilato.fr](mailto:david@pilato.fr) > Hey Isabella, > > I suppose that you are a contractor for a company and that your contract > is to build a plugin, right? > > A river plugin is a plugin. So you respect the contract. > > That said, I would really recommend that you give advices to your client > that it could really be better to have that logic outside elasticsearch > itself. > > My 2 cents. > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > Le 18 novembre 2013 at 11:11:45, Isabella ([isi.huber90@gmail.com](mailto:isi.huber90@gmail.com)//isi.huber90@gmail.com) > a écrit: > > Thanks for your answers. > > Unfortunately I don't know, why it has to be a normal plugin instead of a > river plugin. I only got the instructions, that it has to be a normal > plugin which synchronizes data from postgresql. > > I have to implement a normal plugin, that should read the properties to > connect with the database out of a config file. For instance, how it is > done in the hadoop plugin. > [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop) > > Is there a possibility to configure the database connection properties in > such a config file? > > My second problem is, that I don't know which module I have to use for my > plugin, so that my plugin actually does something when I start ES. Is there > any list of available modules for ES? I only know a few modules like, > RiverModule, AnalysisModule, ScriptModule and RestModule, but I don't think > that these modules are the right for my plugin. Another problem is how can > I address my plugin? Do I need a mapping to say ES that my plugin should do > something? > > I am sorry about my stupid and mabye a litte bit confusing questions. But > my problem is, that I don't know how to initialize my plugin. If it would > be a river plugin, I have to define a mapping with a type, that is also > defined in the java code, when the riverModule is registered, am I right? > But if I have a normal plugin I don't know which module I have to take or > which modules are available. > > I really would appreciate any help. > > Thanks a lot. > > Best Regards, > > Isabella > > Am Donnerstag, 14. November 2013 21:16:25 UTC+1 schrieb Itamar > Syn-Hershko: > > > We like to call this a "shoveller" (or shoveler, depends on your locale), > > as in "shovelling data into ES" > > > > On Thu, Nov 14, 2013 at 4:56 PM, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > > > Question is "why a plugin?" > > > > > > It does not bring so much value to elasticsearch internals. > > > > > > I would start with a simple Main() class (let's say a batch for > > > example). > > > > > > ``` > > > -- > > > > > > ``` > > > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > > > > > Le 14 novembre 2013 at 13:27:52, Isabella ([isi.h...@gmail.com](mailto:isi.h...@gmail.com)) a écrit: > > > > > > Hello! > > > > > > I have to make an plugin which indexes data from my postgresql database, > > > but it should not be a River Plugin. My problem is, that I don't know how > > > to let my plugin know, that I have created a mapping, that defines my jdbc > > > driver, url, password etc. and how to get those properties to connect to my > > > database in my plugin java code? > > > > > > Does anyone know if I can define those jdbc properties in a mapping and > > > read it out or do I have to do it in another way? > > > > > > I really would appreciate your help! > > > Thanks a lot. > > > > > > ## Regards Isabella > > > > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/FQO7TrbejNY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/FQO7TrbejNY/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Isabella](https://avatars.discourse-cdn.com/v4/letter/i/ecb155/32.png) [@Isabella](https://discuss.elastic.co/u/Isabella)\ **Post date:** [November 25, 2013, 9:28am UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/8 "2013-11-25T09:28:41Z")
Thanks a lot for your advice David. I can make a river plugin 🙂 But now I have the problem that I can't create the index for my river plugin. To create my index I do this: client.admin().indices().prepareCreate(indexName).execute().actionGet();. My indexName is "testindex" from my mapping: curl -XPUT 'localhost:9200/\_river/testtype/\_meta' -d '{ "type" : "sync", "sync" : { "driver" : "org.postgresql.Driver", "url" : "jdbc:postgresql://localhost:5432/mydb", "user" : "postgres", "password" : "es", "sql" : "SELECT \* FROM documents" }, "index" : { "index" : "testindex", "type" : "testtype" } }' But I am getting the following error: [2013-11-25 10:03:46,499][WARN][river] [Hephaestus] failed to create river [sync][testtype] java.lang.NullPointerException at org.elasticsearch.plugin.syncdocs.SyncRiver.start(SyncRiver.java:103) at org.elasticsearch.river.RiversService.createRiver(RiversService.java:146) at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:270) at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:264) at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:93) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Do I have to create the river before I have to create the index? I have looked at a view other river plugins and I didn't find some code, where the river is created. Does anyone know if I have to create an index for the river or if I have to take another index instead of my "testindex" to create my index? Thanks a lot for your help. Best regards, Isabella Am Montag, 18. November 2013 12:10:24 UTC+1 schrieb David Pilato: > Hey Isabella, > > I suppose that you are a contractor for a company and that your contract > is to build a plugin, right? > > A river plugin is a plugin. So you respect the contract. > > That said, I would really recommend that you give advices to your client > that it could really be better to have that logic outside elasticsearch > itself. > > My 2 cents. > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X\_lhRIhUzuSZGsaSqA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA) > | @elasticsearchfr[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA) > > Le 18 novembre 2013 at 11:11:45, Isabella ([isi.h...@gmail.com](mailto:isi.h...@gmail.com)\) > a écrit: > > Thanks for your answers. > > Unfortunately I don't know, why it has to be a normal plugin instead of a > river plugin. I only got the instructions, that it has to be a normal > plugin which synchronizes data from postgresql. > > I have to implement a normal plugin, that should read the properties to > connect with the database out of a config file. For instance, how it is > done in the hadoop plugin. > [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)[https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-hadoop&sa=D&sntz=1&usg=AFQjCNFnjr6yHaqjxjB3fDgUyB3FhD10vg](https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-hadoop&sa=D&sntz=1&usg=AFQjCNFnjr6yHaqjxjB3fDgUyB3FhD10vg) > > Is there a possibility to configure the database connection properties in > such a config file? > > My second problem is, that I don't know which module I have to use for my > plugin, so that my plugin actually does something when I start ES. Is there > any list of available modules for ES? I only know a few modules like, > RiverModule, AnalysisModule, ScriptModule and RestModule, but I don't think > that these modules are the right for my plugin. Another problem is how can > I address my plugin? Do I need a mapping to say ES that my plugin should do > something? > > I am sorry about my stupid and mabye a litte bit confusing questions. But > my problem is, that I don't know how to initialize my plugin. If it would > be a river plugin, I have to define a mapping with a type, that is also > defined in the java code, when the riverModule is registered, am I right? > But if I have a normal plugin I don't know which module I have to take or > which modules are available. > > I really would appreciate any help. > > Thanks a lot. > > Best Regards, > > Isabella > > Am Donnerstag, 14. November 2013 21:16:25 UTC+1 schrieb Itamar > Syn-Hershko: > > > We like to call this a "shoveller" (or shoveler, depends on your locale), > > as in "shovelling data into ES" > > > > On Thu, Nov 14, 2013 at 4:56 PM, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > > > Question is "why a plugin?" > > > > > > It does not bring so much value to elasticsearch internals. > > > > > > I would start with a simple Main() class (let's say a batch for > > > example). > > > > > > ``` > > > -- > > > > > > ``` > > > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > > > @dadoonet[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X\_lhRIhUzuSZGsaSqA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA) > > > | @elasticsearchfr[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA) > > > > > > Le 14 novembre 2013 at 13:27:52, Isabella ([isi.h...@gmail.com](mailto:isi.h...@gmail.com)) a écrit: > > > > > > Hello! > > > > > > I have to make an plugin which indexes data from my postgresql database, > > > but it should not be a River Plugin. My problem is, that I don't know how > > > to let my plugin know, that I have created a mapping, that defines my jdbc > > > driver, url, password etc. and how to get those properties to connect to my > > > database in my plugin java code? > > > > > > Does anyone know if I can define those jdbc properties in a mapping and > > > read it out or do I have to do it in another way? > > > > > > I really would appreciate your help! > > > Thanks a lot. > > > > > > ## Regards Isabella > > > > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 25, 2013, 2:03pm UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/9 "2013-11-25T14:03:21Z")
Not sure why you do not use the JDBC river source code as a starting point.... it can save some time to get things started. Without having a look at your code, it is impossible to help, the NPE might not be related to the river at all. No, you don't have to create a river before an index. A river can decide on its own whether to let the index get created automatically or not. JDBC river can do both, either creating a new index, or using an existing index. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Isabella](https://avatars.discourse-cdn.com/v4/letter/i/ecb155/32.png) [@Isabella](https://discuss.elastic.co/u/Isabella)\ **Post date:** [November 25, 2013, 2:46pm UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/10 "2013-11-25T14:46:42Z")
Thanks for your answer. I do take the JDBC river source code as a starting point and I found my error. I forgot to add the client in my constructor in my river class, that was my NPE. Now it works. Thanks! Isabella 2013/11/25 [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com) > Not sure why you do not use the JDBC river source code as a starting > point.... it can save some time to get things started. > > Without having a look at your code, it is impossible to help, the NPE > might not be related to the river at all. > > No, you don't have to create a river before an index. A river can decide > on its own whether to let the index get created automatically or not. JDBC > river can do both, either creating a new index, or using an existing index. > > Jörg > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/FQO7TrbejNY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/FQO7TrbejNY/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:04am UTC](https://discuss.elastic.co/t/index-documents-out-of-postgresql-database-but-without-a-river-plugin/14405/11 "2017-07-06T02:04:59Z")