Index documents out of postgresql database but without a river plugin

Elastic Stack Elasticsearch
1

Hello!

I have to make an plugin which indexes data from my postgresql database,
but it should not be a River Plugin. My problem is, that I don't know how
to let my plugin know, that I have created a mapping, that defines my jdbc
driver, url, password etc. and how to get those properties to connect to my
database in my plugin java code?

Does anyone know if I can define those jdbc properties in a mapping and
read it out or do I have to do it in another way?

I really would appreciate your help!
Thanks a lot.

Regards
Isabella

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Question is "why a plugin?"

It does not bring so much value to elasticsearch internals.

I would start with a simple Main() class (let's say a batch for example).

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 14 novembre 2013 at 13:27:52, Isabella (isi.huber90@gmail.com) a écrit:

Hello!

I have to make an plugin which indexes data from my postgresql database, but it should not be a River Plugin. My problem is, that I don't know how to let my plugin know, that I have created a mapping, that defines my jdbc driver, url, password etc. and how to get those properties to connect to my database in my plugin java code?

Does anyone know if I can define those jdbc properties in a mapping and read it out or do I have to do it in another way?

I really would appreciate your help!
Thanks a lot.

Regards
Isabella

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Probably you want to store the JDBC settings in the cluster state?

That is exactly what a river plugin does :slight_smile:

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

We like to call this a "shoveller" (or shoveler, depends on your locale),
as in "shovelling data into ES"

On Thu, Nov 14, 2013 at 4:56 PM, David Pilato david@pilato.fr wrote:

Question is "why a plugin?"

It does not bring so much value to elasticsearch internals.

I would start with a simple Main() class (let's say a batch for example).

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 14 novembre 2013 at 13:27:52, Isabella (isi.huber90@gmail.com//isi.huber90@gmail.com)
a écrit:

Hello!

I have to make an plugin which indexes data from my postgresql database,
but it should not be a River Plugin. My problem is, that I don't know how
to let my plugin know, that I have created a mapping, that defines my jdbc
driver, url, password etc. and how to get those properties to connect to my
database in my plugin java code?

Does anyone know if I can define those jdbc properties in a mapping and
read it out or do I have to do it in another way?

I really would appreciate your help!
Thanks a lot.

Regards
Isabella

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Thanks for your answers.

Unfortunately I don't know, why it has to be a normal plugin instead of a
river plugin. I only got the instructions, that it has to be a normal
plugin which synchronizes data from postgresql.

I have to implement a normal plugin, that should read the properties to
connect with the database out of a config file. For instance, how it is
done in the hadoop plugin.

Is there a possibility to configure the database connection properties in
such a config file?

My second problem is, that I don't know which module I have to use for my
plugin, so that my plugin actually does something when I start ES. Is there
any list of available modules for ES? I only know a few modules like,
RiverModule, AnalysisModule, ScriptModule and RestModule, but I don't think
that these modules are the right for my plugin. Another problem is how can
I address my plugin? Do I need a mapping to say ES that my plugin should do
something?

I am sorry about my stupid and mabye a litte bit confusing questions. But
my problem is, that I don't know how to initialize my plugin. If it would
be a river plugin, I have to define a mapping with a type, that is also
defined in the java code, when the riverModule is registered, am I right?
But if I have a normal plugin I don't know which module I have to take or
which modules are available.

I really would appreciate any help.

Thanks a lot.

Best Regards,

Isabella

Am Donnerstag, 14. November 2013 21:16:25 UTC+1 schrieb Itamar Syn-Hershko:

We like to call this a "shoveller" (or shoveler, depends on your locale),
as in "shovelling data into ES"

On Thu, Nov 14, 2013 at 4:56 PM, David Pilato <da...@pilato.fr<javascript:>

wrote:

Question is "why a plugin?"

It does not bring so much value to elasticsearch internals.

I would start with a simple Main() class (let's say a batch for example).

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 14 novembre 2013 at 13:27:52, Isabella (isi.h...@gmail.com<javascript:>)
a écrit:

Hello!

I have to make an plugin which indexes data from my postgresql database,
but it should not be a River Plugin. My problem is, that I don't know how
to let my plugin know, that I have created a mapping, that defines my jdbc
driver, url, password etc. and how to get those properties to connect to my
database in my plugin java code?

Does anyone know if I can define those jdbc properties in a mapping and
read it out or do I have to do it in another way?

I really would appreciate your help!
Thanks a lot.

Regards
Isabella

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Hey Isabella,

I suppose that you are a contractor for a company and that your contract is to build a plugin, right?

A river plugin is a plugin. So you respect the contract.

That said, I would really recommend that you give advices to your client that it could really be better to have that logic outside elasticsearch itself.

My 2 cents.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 18 novembre 2013 at 11:11:45, Isabella (isi.huber90@gmail.com) a écrit:

Thanks for your answers.

Unfortunately I don't know, why it has to be a normal plugin instead of a river plugin. I only got the instructions, that it has to be a normal plugin which synchronizes data from postgresql.
I have to implement a normal plugin, that should read the properties to connect with the database out of a config file. For instance, how it is done in the hadoop plugin. https://github.com/elasticsearch/elasticsearch-hadoop
Is there a possibility to configure the database connection properties in such a config file?

My second problem is, that I don't know which module I have to use for my plugin, so that my plugin actually does something when I start ES. Is there any list of available modules for ES? I only know a few modules like, RiverModule, AnalysisModule, ScriptModule and RestModule, but I don't think that these modules are the right for my plugin. Another problem is how can I address my plugin? Do I need a mapping to say ES that my plugin should do something?

I am sorry about my stupid and mabye a litte bit confusing questions. But my problem is, that I don't know how to initialize my plugin. If it would be a river plugin, I have to define a mapping with a type, that is also defined in the java code, when the riverModule is registered, am I right? But if I have a normal plugin I don't know which module I have to take or which modules are available.

I really would appreciate any help.
Thanks a lot.

Best Regards,
Isabella

Am Donnerstag, 14. November 2013 21:16:25 UTC+1 schrieb Itamar Syn-Hershko:
We like to call this a "shoveller" (or shoveler, depends on your locale), as in "shovelling data into ES"

On Thu, Nov 14, 2013 at 4:56 PM, David Pilato da...@pilato.fr wrote:
Question is "why a plugin?"

It does not bring so much value to elasticsearch internals.

I would start with a simple Main() class (let's say a batch for example).

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 14 novembre 2013 at 13:27:52, Isabella (isi.h...@gmail.com) a écrit:

Hello!

I have to make an plugin which indexes data from my postgresql database, but it should not be a River Plugin. My problem is, that I don't know how to let my plugin know, that I have created a mapping, that defines my jdbc driver, url, password etc. and how to get those properties to connect to my database in my plugin java code?

Does anyone know if I can define those jdbc properties in a mapping and read it out or do I have to do it in another way?

I really would appreciate your help!
Thanks a lot.

Regards
Isabella

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hey David!

Thanks a lot for your answer and yes your are right with that.
Ok I will do that.

Thank you.
Isabella

2013/11/18 David Pilato david@pilato.fr

Hey Isabella,

I suppose that you are a contractor for a company and that your contract
is to build a plugin, right?

A river plugin is a plugin. So you respect the contract.

That said, I would really recommend that you give advices to your client
that it could really be better to have that logic outside elasticsearch
itself.

My 2 cents.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 18 novembre 2013 at 11:11:45, Isabella (isi.huber90@gmail.com//isi.huber90@gmail.com)
a écrit:

Thanks for your answers.

Unfortunately I don't know, why it has to be a normal plugin instead of a
river plugin. I only got the instructions, that it has to be a normal
plugin which synchronizes data from postgresql.

I have to implement a normal plugin, that should read the properties to
connect with the database out of a config file. For instance, how it is
done in the hadoop plugin.
GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop

Is there a possibility to configure the database connection properties in
such a config file?

My second problem is, that I don't know which module I have to use for my
plugin, so that my plugin actually does something when I start ES. Is there
any list of available modules for ES? I only know a few modules like,
RiverModule, AnalysisModule, ScriptModule and RestModule, but I don't think
that these modules are the right for my plugin. Another problem is how can
I address my plugin? Do I need a mapping to say ES that my plugin should do
something?

I am sorry about my stupid and mabye a litte bit confusing questions. But
my problem is, that I don't know how to initialize my plugin. If it would
be a river plugin, I have to define a mapping with a type, that is also
defined in the java code, when the riverModule is registered, am I right?
But if I have a normal plugin I don't know which module I have to take or
which modules are available.

I really would appreciate any help.

Thanks a lot.

Best Regards,

Isabella

Am Donnerstag, 14. November 2013 21:16:25 UTC+1 schrieb Itamar
Syn-Hershko:

We like to call this a "shoveller" (or shoveler, depends on your locale),
as in "shovelling data into ES"

On Thu, Nov 14, 2013 at 4:56 PM, David Pilato da...@pilato.fr wrote:

Question is "why a plugin?"

It does not bring so much value to elasticsearch internals.

I would start with a simple Main() class (let's say a batch for
example).

 --

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 14 novembre 2013 at 13:27:52, Isabella (isi.h...@gmail.com) a écrit:

Hello!

I have to make an plugin which indexes data from my postgresql database,
but it should not be a River Plugin. My problem is, that I don't know how
to let my plugin know, that I have created a mapping, that defines my jdbc
driver, url, password etc. and how to get those properties to connect to my
database in my plugin java code?

Does anyone know if I can define those jdbc properties in a mapping and
read it out or do I have to do it in another way?

I really would appreciate your help!
Thanks a lot.

Regards
Isabella

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/FQO7TrbejNY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Thanks a lot for your advice David.

I can make a river plugin :slight_smile:

But now I have the problem that I can't create the index for my river
plugin.
To create my index I do this:

client.admin().indices().prepareCreate(indexName).execute().actionGet();.

My indexName is "testindex" from my mapping:

curl -XPUT 'localhost:9200/_river/testtype/_meta' -d '{

"type" : "sync",

"sync" : {

"driver" : "org.postgresql.Driver",

"url" : "jdbc:postgresql://localhost:5432/mydb",

"user" : "postgres",

"password" : "es",

"sql" : "SELECT * FROM documents"

},
"index" : {

"index" : "testindex",

"type" : "testtype"

}

}'

But I am getting the following error:

[2013-11-25 10:03:46,499][WARN ][river ] [Hephaestus]
failed to create river [sync][testtype]
java.lang.NullPointerException
at org.elasticsearch.plugin.syncdocs.SyncRiver.start(SyncRiver.java:103)
at org.elasticsearch.river.RiversService.createRiver(RiversService.java:146)
at
org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:270)
at
org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:264)
at
org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:93)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

Do I have to create the river before I have to create the index? I have
looked at a view other river plugins and I didn't find some code, where the
river is created.
Does anyone know if I have to create an index for the river or if I have to
take another index instead of my "testindex" to create my index?

Thanks a lot for your help.

Best regards,
Isabella

Am Montag, 18. November 2013 12:10:24 UTC+1 schrieb David Pilato:

Hey Isabella,

I suppose that you are a contractor for a company and that your contract
is to build a plugin, right?

A river plugin is a plugin. So you respect the contract.

That said, I would really recommend that you give advices to your client
that it could really be better to have that logic outside elasticsearch
itself.

My 2 cents.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonethttps://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA
| @elasticsearchfrhttps://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA

Le 18 novembre 2013 at 11:11:45, Isabella (isi.h...@gmail.com<javascript:>)
a écrit:

Thanks for your answers.

Unfortunately I don't know, why it has to be a normal plugin instead of a
river plugin. I only got the instructions, that it has to be a normal
plugin which synchronizes data from postgresql.

I have to implement a normal plugin, that should read the properties to
connect with the database out of a config file. For instance, how it is
done in the hadoop plugin.
GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoophttps://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-hadoop&sa=D&sntz=1&usg=AFQjCNFnjr6yHaqjxjB3fDgUyB3FhD10vg

Is there a possibility to configure the database connection properties in
such a config file?

My second problem is, that I don't know which module I have to use for my
plugin, so that my plugin actually does something when I start ES. Is there
any list of available modules for ES? I only know a few modules like,
RiverModule, AnalysisModule, ScriptModule and RestModule, but I don't think
that these modules are the right for my plugin. Another problem is how can
I address my plugin? Do I need a mapping to say ES that my plugin should do
something?

I am sorry about my stupid and mabye a litte bit confusing questions. But
my problem is, that I don't know how to initialize my plugin. If it would
be a river plugin, I have to define a mapping with a type, that is also
defined in the java code, when the riverModule is registered, am I right?
But if I have a normal plugin I don't know which module I have to take or
which modules are available.

I really would appreciate any help.

Thanks a lot.

Best Regards,

Isabella

Am Donnerstag, 14. November 2013 21:16:25 UTC+1 schrieb Itamar
Syn-Hershko:

We like to call this a "shoveller" (or shoveler, depends on your locale),
as in "shovelling data into ES"

On Thu, Nov 14, 2013 at 4:56 PM, David Pilato da...@pilato.fr wrote:

Question is "why a plugin?"

It does not bring so much value to elasticsearch internals.

I would start with a simple Main() class (let's say a batch for
example).

 --

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonethttps://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA
| @elasticsearchfrhttps://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA

Le 14 novembre 2013 at 13:27:52, Isabella (isi.h...@gmail.com) a écrit:

Hello!

I have to make an plugin which indexes data from my postgresql database,
but it should not be a River Plugin. My problem is, that I don't know how
to let my plugin know, that I have created a mapping, that defines my jdbc
driver, url, password etc. and how to get those properties to connect to my
database in my plugin java code?

Does anyone know if I can define those jdbc properties in a mapping and
read it out or do I have to do it in another way?

I really would appreciate your help!
Thanks a lot.

Regards
Isabella

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Not sure why you do not use the JDBC river source code as a starting
point.... it can save some time to get things started.

Without having a look at your code, it is impossible to help, the NPE might
not be related to the river at all.

No, you don't have to create a river before an index. A river can decide on
its own whether to let the index get created automatically or not. JDBC
river can do both, either creating a new index, or using an existing index.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

Thanks for your answer.

I do take the JDBC river source code as a starting point and I found my
error.
I forgot to add the client in my constructor in my river class, that was my
NPE.
Now it works.

Thanks!
Isabella

2013/11/25 joergprante@gmail.com joergprante@gmail.com

Not sure why you do not use the JDBC river source code as a starting
point.... it can save some time to get things started.

Without having a look at your code, it is impossible to help, the NPE
might not be related to the river at all.

No, you don't have to create a river before an index. A river can decide
on its own whether to let the index get created automatically or not. JDBC
river can do both, either creating a new index, or using an existing index.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/FQO7TrbejNY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.