How to do incremental indexing in ElasticSearch?

Daniel_Guo · February 21, 2014, 2:04am

Here is my problem:

I query some data from MySql, and then index those data to ElasticSearch.

While the data in MySql is updating (update and insert) all the time, so I
have to update ElasticSearch index accordingly.

I cannot afford to do full index (data is huge), and I should not do that
either. Does ElasticSearch provide any mechanism to simulate incremental
indexing? Or I have to
implement it by myself when importing data from MySql?

Any ideas? Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · February 21, 2014, 6:32am

Elasticsearch will index or update any document you will send to it.
So get the delta on your side and send documents you want to update to elasticsearch.

Did I misunderstand the question?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 21 févr. 2014 à 03:04, Daniel Guo daniel5hbs@gmail.com a écrit :

Here is my problem:

I query some data from MySql, and then index those data to ElasticSearch.

While the data in MySql is updating (update and insert) all the time, so I have to update ElasticSearch index accordingly.

I cannot afford to do full index (data is huge), and I should not do that either. Does ElasticSearch provide any mechanism to simulate incremental indexing? Or I have to
implement it by myself when importing data from MySql?

Any ideas? Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/FCAFD092-1594-4C6D-9E67-698A6C208DC9%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

Daniel_Guo · February 21, 2014, 9:12am

David, thanks for your answer. You mean that I have to control what to send
to elasticsearch by myself ?!

I heard that solr can configure delta index, so I'm just looking for a
similar mechanism in elasicsearch.

Thank you one more time.

On Friday, February 21, 2014 2:32:42 PM UTC+8, David Pilato wrote:

Elasticsearch will index or update any document you will send to it.
So get the delta on your side and send documents you want to update to
elasticsearch.

Did I misunderstand the question?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 21 févr. 2014 à 03:04, Daniel Guo <danie...@gmail.com <javascript:>> a
écrit :

Here is my problem:

I query some data from MySql, and then index those data to Elasticsearch.

While the data in MySql is updating (update and insert) all the time, so I
have to update Elasticsearch index accordingly.

I cannot afford to do full index (data is huge), and I should not do that
either. Does Elasticsearch provide any mechanism to simulate incremental
indexing? Or I have to
implement it by myself when importing data from MySql?

Any ideas? Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fc45880d-9bb4-4155-9713-240b78684284%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · February 21, 2014, 10:29am

Yes.

But you can look at JDBC river although I won't use it in a production env such as all other rivers.

I prefer to push docs from my legacy application (in real time that is)

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 21 févr. 2014 à 10:12, Daniel Guo daniel5hbs@gmail.com a écrit :

David, thanks for your answer. You mean that I have to control what to send to elasticsearch by myself ?!

I heard that solr can configure delta index, so I'm just looking for a similar mechanism in elasicsearch.

Thank you one more time.

On Friday, February 21, 2014 2:32:42 PM UTC+8, David Pilato wrote:

Elasticsearch will index or update any document you will send to it.
So get the delta on your side and send documents you want to update to elasticsearch.

Did I misunderstand the question?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 21 févr. 2014 à 03:04, Daniel Guo danie...@gmail.com a écrit :

Here is my problem:

I query some data from MySql, and then index those data to Elasticsearch.

While the data in MySql is updating (update and insert) all the time, so I have to update Elasticsearch index accordingly.

I cannot afford to do full index (data is huge), and I should not do that either. Does Elasticsearch provide any mechanism to simulate incremental indexing? Or I have to
implement it by myself when importing data from MySql?

Any ideas? Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fc45880d-9bb4-4155-9713-240b78684284%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/B07787A3-FF06-458D-96E2-74C14B7026CE%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

Daniel_Guo · February 21, 2014, 10:32am

Ok, got it. Thank you.

On Friday, February 21, 2014 6:29:35 PM UTC+8, David Pilato wrote:

Yes.

But you can look at JDBC river although I won't use it in a production env
such as all other rivers.

I prefer to push docs from my legacy application (in real time that is)

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 21 févr. 2014 à 10:12, Daniel Guo <danie...@gmail.com <javascript:>> a
écrit :

David, thanks for your answer. You mean that I have to control what to
send to elasticsearch by myself ?!

I heard that solr can configure delta index, so I'm just looking for a
similar mechanism in elasicsearch.

Thank you one more time.

On Friday, February 21, 2014 2:32:42 PM UTC+8, David Pilato wrote:

Elasticsearch will index or update any document you will send to it.
So get the delta on your side and send documents you want to update to
elasticsearch.

Did I misunderstand the question?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 21 févr. 2014 à 03:04, Daniel Guo danie...@gmail.com a écrit :

Here is my problem:

I query some data from MySql, and then index those data to Elasticsearch.

While the data in MySql is updating (update and insert) all the time, so
I have to update Elasticsearch index accordingly.

I cannot afford to do full index (data is huge), and I should not do that
either. Does Elasticsearch provide any mechanism to simulate incremental
indexing? Or I have to
implement it by myself when importing data from MySql?

Any ideas? Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8746d25b-078f-4f08-aa6a-08accb15e4af%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fc45880d-9bb4-4155-9713-240b78684284%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7cf3a4aa-7b98-4a66-bd3f-2eae7341ecda%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.