# How to do incremental indexing in ElasticSearch? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 21, 2014, 2:04am UTC](https://discuss.elastic.co/t/how-to-do-incremental-indexing-in-elasticsearch/15937 "2014-02-21T02:04:32Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Daniel\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_guo/32/1725_2.png) [@Daniel\_Guo](https://discuss.elastic.co/u/Daniel_Guo)\ **Post date:** [February 21, 2014, 2:04am UTC](https://discuss.elastic.co/t/how-to-do-incremental-indexing-in-elasticsearch/15937/1 "2014-02-21T02:04:32Z")
Here is my problem: I query some data from MySql, and then index those data to ElasticSearch. While the data in MySql is updating (update and insert) all the time, so I have to update ElasticSearch index accordingly. I cannot afford to do full index (data is huge), and I should not do that either. Does ElasticSearch provide any mechanism to simulate incremental indexing? Or I have to implement it by myself when importing data from MySql? Any ideas? Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 21, 2014, 6:32am UTC](https://discuss.elastic.co/t/how-to-do-incremental-indexing-in-elasticsearch/15937/2 "2014-02-21T06:32:42Z")
Elasticsearch will index or update any document you will send to it. So get the delta on your side and send documents you want to update to elasticsearch. Did I misunderstand the question? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 21 févr. 2014 à 03:04, Daniel Guo [daniel5hbs@gmail.com](mailto:daniel5hbs@gmail.com) a écrit : Here is my problem: I query some data from MySql, and then index those data to ElasticSearch. While the data in MySql is updating (update and insert) all the time, so I have to update ElasticSearch index accordingly. I cannot afford to do full index (data is huge), and I should not do that either. Does ElasticSearch provide any mechanism to simulate incremental indexing? Or I have to implement it by myself when importing data from MySql? Any ideas? Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/FCAFD092-1594-4C6D-9E67-698A6C208DC9%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/FCAFD092-1594-4C6D-9E67-698A6C208DC9%40pilato.fr). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Daniel\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_guo/32/1725_2.png) [@Daniel\_Guo](https://discuss.elastic.co/u/Daniel_Guo)\ **Post date:** [February 21, 2014, 9:12am UTC](https://discuss.elastic.co/t/how-to-do-incremental-indexing-in-elasticsearch/15937/3 "2014-02-21T09:12:46Z")
David, thanks for your answer. You mean that I have to control what to send to elasticsearch by myself ?! I heard that solr can configure delta index, so I'm just looking for a similar mechanism in elasicsearch. Thank you one more time. On Friday, February 21, 2014 2:32:42 PM UTC+8, David Pilato wrote: > Elasticsearch will index or update any document you will send to it. > So get the delta on your side and send documents you want to update to > elasticsearch. > > Did I misunderstand the question? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 21 févr. 2014 à 03:04, Daniel Guo \<[danie...@gmail.com](mailto:danie...@gmail.com) \\> a > écrit : > > Here is my problem: > > I query some data from MySql, and then index those data to Elasticsearch. > > While the data in MySql is updating (update and insert) all the time, so I > have to update Elasticsearch index accordingly. > > I cannot afford to do full index (data is huge), and I should not do that > either. Does Elasticsearch provide any mechanism to simulate incremental > indexing? Or I have to > implement it by myself when importing data from MySql? > > Any ideas? Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fc45880d-9bb4-4155-9713-240b78684284%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fc45880d-9bb4-4155-9713-240b78684284%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 21, 2014, 10:29am UTC](https://discuss.elastic.co/t/how-to-do-incremental-indexing-in-elasticsearch/15937/4 "2014-02-21T10:29:35Z")
Yes. But you can look at JDBC river although I won't use it in a production env such as all other rivers. I prefer to push docs from my legacy application (in real time that is) -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 21 févr. 2014 à 10:12, Daniel Guo [daniel5hbs@gmail.com](mailto:daniel5hbs@gmail.com) a écrit : > David, thanks for your answer. You mean that I have to control what to send to elasticsearch by myself ?! > > I heard that solr can configure delta index, so I'm just looking for a similar mechanism in elasicsearch. > > Thank you one more time. > > On Friday, February 21, 2014 2:32:42 PM UTC+8, David Pilato wrote: > > > Elasticsearch will index or update any document you will send to it. > > So get the delta on your side and send documents you want to update to elasticsearch. > > > > Did I misunderstand the question? > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 21 févr. 2014 à 03:04, Daniel Guo [danie...@gmail.com](mailto:danie...@gmail.com) a écrit : > > > > Here is my problem: > > > > I query some data from MySql, and then index those data to Elasticsearch. > > > > While the data in MySql is updating (update and insert) all the time, so I have to update Elasticsearch index accordingly. > > > > I cannot afford to do full index (data is huge), and I should not do that either. Does Elasticsearch provide any mechanism to simulate incremental indexing? Or I have to > > implement it by myself when importing data from MySql? > > > > Any ideas? Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fc45880d-9bb4-4155-9713-240b78684284%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fc45880d-9bb4-4155-9713-240b78684284%40googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/B07787A3-FF06-458D-96E2-74C14B7026CE%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/B07787A3-FF06-458D-96E2-74C14B7026CE%40pilato.fr). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Daniel\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_guo/32/1725_2.png) [@Daniel\_Guo](https://discuss.elastic.co/u/Daniel_Guo)\ **Post date:** [February 21, 2014, 10:32am UTC](https://discuss.elastic.co/t/how-to-do-incremental-indexing-in-elasticsearch/15937/5 "2014-02-21T10:32:23Z")
Ok, got it. Thank you. On Friday, February 21, 2014 6:29:35 PM UTC+8, David Pilato wrote: > Yes. > > But you can look at JDBC river although I won't use it in a production env > such as all other rivers. > > I prefer to push docs from my legacy application (in real time that is) > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 21 févr. 2014 à 10:12, Daniel Guo \<[danie...@gmail.com](mailto:danie...@gmail.com) \\> a > écrit : > > David, thanks for your answer. You mean that I have to control what to > send to elasticsearch by myself ?! > > I heard that solr can configure delta index, so I'm just looking for a > similar mechanism in elasicsearch. > > Thank you one more time. > > On Friday, February 21, 2014 2:32:42 PM UTC+8, David Pilato wrote: > > > Elasticsearch will index or update any document you will send to it. > > So get the delta on your side and send documents you want to update to > > elasticsearch. > > > > Did I misunderstand the question? > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 21 févr. 2014 à 03:04, Daniel Guo [danie...@gmail.com](mailto:danie...@gmail.com) a écrit : > > > > Here is my problem: > > > > I query some data from MySql, and then index those data to Elasticsearch. > > > > While the data in MySql is updating (update and insert) all the time, so > > I have to update Elasticsearch index accordingly. > > > > I cannot afford to do full index (data is huge), and I should not do that > > either. Does Elasticsearch provide any mechanism to simulate incremental > > indexing? Or I have to > > implement it by myself when importing data from MySql? > > > > Any ideas? Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/fc45880d-9bb4-4155-9713-240b78684284%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fc45880d-9bb4-4155-9713-240b78684284%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7cf3a4aa-7b98-4a66-bd3f-2eae7341ecda%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7cf3a4aa-7b98-4a66-bd3f-2eae7341ecda%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:48am UTC](https://discuss.elastic.co/t/how-to-do-incremental-indexing-in-elasticsearch/15937/6 "2017-07-06T01:48:26Z")