Chunk-wise incremental data feed into ES jdbc feeder/river

Elastic Stack Elasticsearch
1

does ES jdbc river/feeder have a mechanism to feed data from database into
ES via jdbc incrementally in a stepwise function?
I have billions of records to be imported into ES from mysql and a select *
from my table would take a long time and be interrupted due to db timeout.
I need to do select * from my_table where my_id>start_id and my_id<end_id
and iterate through somehow. feeder/river doesn't seem to provides such
mechanism, or am I missing something?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f8640525-fc46-4dcd-8fbe-a4ce0cd50f78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

JDBC plugin supports MySQL streaming mode out of the box:

If it does not work, I'm available for help to find alternatives.

I would love to know more about the observed DB timeout. The MySQL timeouts
can be changed in configuration, but I am unsure if streaming mode can
result in timeouts - that would be a defect on MySQL JDBC driver level.

Jörg

On Sun, Apr 5, 2015 at 12:18 AM, Jonathan Zhang topcoder1@gmail.com wrote:

does ES jdbc river/feeder have a mechanism to feed data from database into
ES via jdbc incrementally in a stepwise function?
I have billions of records to be imported into ES from mysql and a select

  • from my table would take a long time and be interrupted due to db
    timeout. I need to do select * from my_table where my_id>start_id and
    my_id<end_id and iterate through somehow. feeder/river doesn't seem to
    provides such mechanism, or am I missing something?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f8640525-fc46-4dcd-8fbe-a4ce0cd50f78%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f8640525-fc46-4dcd-8fbe-a4ce0cd50f78%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEzaS3y97PBSr%3DHRnsvYvPSNwU_%2Bv6sb%2BVXX%2BimQxf7Hg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.