JDBC River Polling Doesn't Work

Eric_4 · March 28, 2013, 10:02am

Hi,

I started a river to an index (it has only firstname and lastname) with a
command like below.

curl -XPUT 'localhost:9200/_river/test_jdbc_name_river/_meta' -d '{
"type" : "jdbc",
"jdbc" : {
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://localhost:3306/testdb",
"user" : "user",
"password" : "user",
"sql" : "select id as _id, firstname, lastname from person",
"poll" : "1m"
},
"index" : {
"index" : "test_river",
"type" : "person"
}
}'

It seems that the polling doesn't happen with one minute intervals.
Additionally it deleted 800 records under housekeeping as mentioned in the
log (the view contains 39819 records but the index is having only 39019). I
changed a row in a db table and waited. But the change was not propagated.

How can I make sure the sync happens properly?

Have I mentioned the poll time correctly? I tried with "30s" but it didn't
work either.

() I am referring to the river implementation at

() The "person" is actually a view of several tables. So, adding timestamp
details is not possible.
(*) Didn't attach the "elasticsearch.log" since it made issues in posting.
.
Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Eric_4 · March 28, 2013, 10:05am

Following is the last part of the log.

[2013-03-28 17:13:04,533][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] new bulk
[406] of [100 items], 5 outstanding bulk requests
[2013-03-28 17:13:04,534][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] new bulk
[407] of [100 items], 6 outstanding bulk requests
[2013-03-28 17:13:04,534][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow]
housekeeping done, 800 documents deleted, took 42 ms
[2013-03-28 17:13:04,535][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] bulk [406]
success [100 items] [2ms]
[2013-03-28 17:13:04,535][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] bulk [407]
success [100 items] [1ms]
[2013-03-28 17:13:04,536][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] bulk [401]
success [100 items] [7ms]
[2013-03-28 17:13:04,539][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] bulk [405]
success [100 items] [7ms]
[2013-03-28 17:13:04,541][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] bulk [400]
success [100 items] [13ms]
[2013-03-28 17:13:04,541][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] bulk [403]
success [100 items] [10ms]
[2013-03-28 17:13:05,337][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] new bulk
[408] of [71 items], 1 outstanding bulk requests
[2013-03-28 17:13:05,338][INFO
][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth] bulk [408]
success [71 items] [1ms]

On Thursday, March 28, 2013 6:02:36 PM UTC+8, Eric wrote:

Hi,

I started a river to an index (it has only firstname and lastname) with a
command like below.

curl -XPUT 'localhost:9200/_river/test_jdbc_name_river/_meta' -d '{
"type" : "jdbc",
"jdbc" : {
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://localhost:3306/testdb",
"user" : "user",
"password" : "user",
"sql" : "select id as _id, firstname, lastname from person",
"poll" : "1m"
},
"index" : {
"index" : "test_river",
"type" : "person"
}
}'

It seems that the polling doesn't happen with one minute intervals.
Additionally it deleted 800 records under housekeeping as mentioned in the
log (the view contains 39819 records but the index is having only 39019). I
changed a row in a db table and waited. But the change was not propagated.

How can I make sure the sync happens properly?

Have I mentioned the poll time correctly? I tried with "30s" but it didn't
work either.

() I am referring to the river implementation at
GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch
() The "person" is actually a view of several tables. So, adding
timestamp details is not possible.
(*) Didn't attach the "elasticsearch.log" since it made issues in posting.
.
Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · March 28, 2013, 2:48pm

Hi,

thanks for reporting, I will look into it! For tracing the issue I
opened a ticket

github.com/jprante/elasticsearch-jdbc

polling doesn't happen

opened 02:48PM - 28 Mar 13 UTC

closed 07:21AM - 19 May 14 UTC

jprante

I started a river to an index (it has only firstname and lastname) with a comman…d like below. curl -XPUT 'localhost:9200/_river/test_jdbc_name_river/_meta' -d '{ "type" : "jdbc", "jdbc" : { "driver" : "com.mysql.jdbc.Driver", "url" : "jdbc:mysql://localhost:3306/testdb", "user" : "user", "password" : "user", "sql" : "select id as _id, firstname, lastname from person", "poll" : "1m" }, "index" : { "index" : "test_river", "type" : "person" } }' It seems that the polling doesn't happen with one minute intervals. Additionally it deleted 800 records under housekeeping as mentioned in the log (the view contains 39819 records but the index is having only 39019). I changed a row in a db table and waited. But the change was not propagated. How can I make sure the sync happens properly? Have I mentioned the poll time correctly? I tried with "30s" but it didn't work either. (_) I am referring to the river implementation at https://github.com/jprante/elasticsearch-river-jdbc (_) The "person" is actually a view of several tables. So, adding timestamp details is not possible. (*) Didn't attach the "elasticsearch.log" since it made issues in posting. . Thank you.

Jörg

Am 28.03.13 11:02, schrieb Eric:

Hi,

I started a river to an index (it has only firstname and lastname)
with a command like below.

|
curl -XPUT 'localhost:9200/_river/test_jdbc_name_river/_meta'-d '{
"type" : "jdbc",
"jdbc" : {
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://localhost:3306/testdb",
"user" : "user",
"password" : "user",
"sql" : "select id as _id, firstname, lastname from person",
"poll" : "1m"
},
"index" : {
"index" : "test_river",
"type" : "person"
}
}'
|

It seems that the polling doesn't happen with one minute intervals.
Additionally it deleted 800 records under housekeeping as mentioned in
the log (the view contains 39819 records but the index is having
only 39019). I changed a row in a db table and waited. But the change
was not propagated.

How can I make sure the sync happens properly?

Have I mentioned the poll time correctly? I tried with "30s" but it
didn't work either.

() I am referring to the river implementation at
GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch
() The "person" is actually a view of several tables. So, adding
timestamp details is not possible.
(*) Didn't attach the "elasticsearch.log" since it made issues in posting.
.
Thank you.

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.