# Chunk-wise incremental data feed into ES jdbc feeder/river

**URL:** <https://discuss.elastic.co/t/chunk-wise-incremental-data-feed-into-es-jdbc-feeder-river/23097>\
**Category:** Elasticsearch\
**Created:** [April 4, 2015, 10:18pm UTC](https://discuss.elastic.co/t/chunk-wise-incremental-data-feed-into-es-jdbc-feeder-river/23097 "2015-04-04T22:18:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jonathan\_Zhang](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@Jonathan\_Zhang](https://discuss.elastic.co/u/Jonathan_Zhang)\
**Post date:** [April 4, 2015, 10:18pm UTC](https://discuss.elastic.co/t/chunk-wise-incremental-data-feed-into-es-jdbc-feeder-river/23097/1 "2015-04-04T22:18:56Z")

</div>

does ES jdbc river/feeder have a mechanism to feed data from database into  
ES via jdbc incrementally in a stepwise function?  
I have billions of records to be imported into ES from mysql and a select \*  
from my table would take a long time and be interrupted due to db timeout.  
I need to do select \* from my\_table where my\_id\>start\_id and my\_id\<end\_id  
and iterate through somehow. feeder/river doesn't seem to provides such  
mechanism, or am I missing something?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f8640525-fc46-4dcd-8fbe-a4ce0cd50f78%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f8640525-fc46-4dcd-8fbe-a4ce0cd50f78%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [April 5, 2015, 4:49pm UTC](https://discuss.elastic.co/t/chunk-wise-incremental-data-feed-into-es-jdbc-feeder-river/23097/2 "2015-04-05T16:49:11Z")

</div>

JDBC plugin supports MySQL streaming mode out of the box:

> <https://github.com/jprante/elasticsearch-jdbc/issues/520#issuecomment-89789655>
>
> does ES river/feeder have a mechanism to feed data from database into ES via jdb…c incrementally in a stepwise function? 
> I have billions of records to be imported into ES from mysql and a select \\\* from my table would take a long time and be interrupted. I need to do select \\\* from my\_table where my\_id\>start\_id and my\_id\<end\_id and iterate through somehow. feeder/river doesn't seem to provides such mechanism, or am I missing something?

If it does not work, I'm available for help to find alternatives.

I would love to know more about the observed DB timeout. The MySQL timeouts  
can be changed in configuration, but I am unsure if streaming mode can  
result in timeouts - that would be a defect on MySQL JDBC driver level.

Jörg

On Sun, Apr 5, 2015 at 12:18 AM, Jonathan Zhang [topcoder1@gmail.com](mailto:topcoder1@gmail.com) wrote:

> does ES jdbc river/feeder have a mechanism to feed data from database into  
> ES via jdbc incrementally in a stepwise function?  
> I have billions of records to be imported into ES from mysql and a select
> 
> - from my table would take a long time and be interrupted due to db  
> timeout. I need to do select \* from my\_table where my\_id\>start\_id and  
> my\_id\<end\_id and iterate through somehow. feeder/river doesn't seem to  
> provides such mechanism, or am I missing something?
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/f8640525-fc46-4dcd-8fbe-a4ce0cd50f78%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f8640525-fc46-4dcd-8fbe-a4ce0cd50f78%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/f8640525-fc46-4dcd-8fbe-a4ce0cd50f78%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f8640525-fc46-4dcd-8fbe-a4ce0cd50f78%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEzaS3y97PBSr%3DHRnsvYvPSNwU\_%2Bv6sb%2BVXX%2BimQxf7Hg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEzaS3y97PBSr%3DHRnsvYvPSNwU_%2Bv6sb%2BVXX%2BimQxf7Hg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:21am UTC](https://discuss.elastic.co/t/chunk-wise-incremental-data-feed-into-es-jdbc-feeder-river/23097/3 "2017-07-06T00:21:39Z")

</div>