Call _optimize during production work


(Kirill Teplinskiy) #1

Hello!

Can anyone tell is it safe to call _optimize under normal production work?
Will search requests be responded and indexing be performed?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #2

In 99.9% you should not call optimize api and let elasticsearch/Lucene do it for you when needed.

To answer to your question, yes search and index operations will still possible during that time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 14:37, Kirill Teplinskiy tkirill128@gmail.com a écrit :

Hello!

Can anyone tell is it safe to call _optimize under normal production work? Will search requests be responded and indexing be performed?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/BF20D0F9-0BAA-4A23-9303-0F39F63C1F42%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(Kirill Teplinskiy) #3

Thank you, David!

I need to call _optimize by hands to refresh payloads in completion
suggester. This method is recommended here:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/search-suggesters-completion.html.
For some reason _suggest returns old payloads after reindexing on our stage
server. I don't know why, completion suggester on my local instance of
ElasticSearch updates on the fly. Maybe because our stage index is 10Gb
size and contains 10 millions documents and my local index has only 180 000
documents.

On Thursday, May 29, 2014 7:37:24 PM UTC+6, David Pilato wrote:

In 99.9% you should not call optimize api and let elasticsearch/Lucene do
it for you when needed.

To answer to your question, yes search and index operations will still
possible during that time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 14:37, Kirill Teplinskiy <tkiri...@gmail.com<javascript:>>
a écrit :

Hello!

Can anyone tell is it safe to call _optimize under normal production
work? Will search requests be responded and indexing be performed?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/95b0bfcf-38b5-4880-ae50-efaa11aaddb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #4

So you reindex into the same index and not in another clean one?
So you "update" docs, right?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 16:54, Kirill Teplinskiy tkirill128@gmail.com a écrit :

Thank you, David!

I need to call _optimize by hands to refresh payloads in completion suggester. This method is recommended here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/search-suggesters-completion.html. For some reason _suggest returns old payloads after reindexing on our stage server. I don't know why, completion suggester on my local instance of ElasticSearch updates on the fly. Maybe because our stage index is 10Gb size and contains 10 millions documents and my local index has only 180 000 documents.

On Thursday, May 29, 2014 7:37:24 PM UTC+6, David Pilato wrote:
In 99.9% you should not call optimize api and let elasticsearch/Lucene do it for you when needed.

To answer to your question, yes search and index operations will still possible during that time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 14:37, Kirill Teplinskiy tkiri...@gmail.com a écrit :

Hello!

Can anyone tell is it safe to call _optimize under normal production work? Will search requests be responded and indexing be performed?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/95b0bfcf-38b5-4880-ae50-efaa11aaddb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/DA72C917-FE7C-4040-A1DC-3B95ADC7AE3E%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(Kirill Teplinskiy) #5

Yes, exactly. I update docs in the same index.

On Thursday, May 29, 2014 9:10:40 PM UTC+6, David Pilato wrote:

So you reindex into the same index and not in another clean one?
So you "update" docs, right?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 16:54, Kirill Teplinskiy <tkiri...@gmail.com<javascript:>>
a écrit :

Thank you, David!

I need to call _optimize by hands to refresh payloads in completion
suggester. This method is recommended here:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/search-suggesters-completion.html.
For some reason _suggest returns old payloads after reindexing on our stage
server. I don't know why, completion suggester on my local instance of
ElasticSearch updates on the fly. Maybe because our stage index is 10Gb
size and contains 10 millions documents and my local index has only 180 000
documents.

On Thursday, May 29, 2014 7:37:24 PM UTC+6, David Pilato wrote:

In 99.9% you should not call optimize api and let elasticsearch/Lucene do
it for you when needed.

To answer to your question, yes search and index operations will still
possible during that time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 14:37, Kirill Teplinskiy tkiri...@gmail.com a écrit :

Hello!

Can anyone tell is it safe to call _optimize under normal production
work? Will search requests be responded and indexing be performed?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/95b0bfcf-38b5-4880-ae50-efaa11aaddb1%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/95b0bfcf-38b5-4880-ae50-efaa11aaddb1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f830d7a1-8ad8-4390-9860-dc04029c7cd8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #6

So it created new segments and at some point expunges deletes but not all.
TBH, I'd prefer using another index with an alias on top of the older one and at the end, switch the alias to the new one and delete old index.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 17:29, Kirill Teplinskiy tkirill128@gmail.com a écrit :

Yes, exactly. I update docs in the same index.

On Thursday, May 29, 2014 9:10:40 PM UTC+6, David Pilato wrote:
So you reindex into the same index and not in another clean one?
So you "update" docs, right?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 16:54, Kirill Teplinskiy tkiri...@gmail.com a écrit :

Thank you, David!

I need to call _optimize by hands to refresh payloads in completion suggester. This method is recommended here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/search-suggesters-completion.html. For some reason _suggest returns old payloads after reindexing on our stage server. I don't know why, completion suggester on my local instance of ElasticSearch updates on the fly. Maybe because our stage index is 10Gb size and contains 10 millions documents and my local index has only 180 000 documents.

On Thursday, May 29, 2014 7:37:24 PM UTC+6, David Pilato wrote:
In 99.9% you should not call optimize api and let elasticsearch/Lucene do it for you when needed.

To answer to your question, yes search and index operations will still possible during that time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 14:37, Kirill Teplinskiy tkiri...@gmail.com a écrit :

Hello!

Can anyone tell is it safe to call _optimize under normal production work? Will search requests be responded and indexing be performed?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/95b0bfcf-38b5-4880-ae50-efaa11aaddb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f830d7a1-8ad8-4390-9860-dc04029c7cd8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4FD36226-3CDC-4AF2-9327-60C72CD2B196%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(Kirill Teplinskiy) #7

Oh, it is interesting. I really had some suggestions with new payloads and
some with old! Thank you for explain! As I understand, Lucene perform
update operation as delete old document and create new document?

Your idea with alias looks great. I think we will use it for the next full
reindex, thank you again.

On Thursday, May 29, 2014 9:40:36 PM UTC+6, David Pilato wrote:

So it created new segments and at some point expunges deletes but not all.
TBH, I'd prefer using another index with an alias on top of the older one
and at the end, switch the alias to the new one and delete old index.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 17:29, Kirill Teplinskiy <tkiri...@gmail.com<javascript:>>
a écrit :

Yes, exactly. I update docs in the same index.

On Thursday, May 29, 2014 9:10:40 PM UTC+6, David Pilato wrote:

So you reindex into the same index and not in another clean one?
So you "update" docs, right?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 16:54, Kirill Teplinskiy tkiri...@gmail.com a écrit :

Thank you, David!

I need to call _optimize by hands to refresh payloads in completion
suggester. This method is recommended here:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/search-suggesters-completion.html.
For some reason _suggest returns old payloads after reindexing on our stage
server. I don't know why, completion suggester on my local instance of
ElasticSearch updates on the fly. Maybe because our stage index is 10Gb
size and contains 10 millions documents and my local index has only 180 000
documents.

On Thursday, May 29, 2014 7:37:24 PM UTC+6, David Pilato wrote:

In 99.9% you should not call optimize api and let elasticsearch/Lucene
do it for you when needed.

To answer to your question, yes search and index operations will still
possible during that time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 14:37, Kirill Teplinskiy tkiri...@gmail.com a écrit :

Hello!

Can anyone tell is it safe to call _optimize under normal production
work? Will search requests be responded and indexing be performed?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/95b0bfcf-38b5-4880-ae50-efaa11aaddb1%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/95b0bfcf-38b5-4880-ae50-efaa11aaddb1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f830d7a1-8ad8-4390-9860-dc04029c7cd8%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/f830d7a1-8ad8-4390-9860-dc04029c7cd8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2d0e601e-55f6-456c-8901-42e074705d26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #8

exactly. An update is a delete and create behind the scene.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 29 mai 2014 à 17:57:40, Kirill Teplinskiy (tkirill128@gmail.com) a écrit:

Oh, it is interesting. I really had some suggestions with new payloads and some with old! Thank you for explain! As I understand, Lucene perform update operation as delete old document and create new document?

Your idea with alias looks great. I think we will use it for the next full reindex, thank you again.

On Thursday, May 29, 2014 9:40:36 PM UTC+6, David Pilato wrote:
So it created new segments and at some point expunges deletes but not all.
TBH, I'd prefer using another index with an alias on top of the older one and at the end, switch the alias to the new one and delete old index.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 17:29, Kirill Teplinskiy tkiri...@gmail.com a écrit :

Yes, exactly. I update docs in the same index.

On Thursday, May 29, 2014 9:10:40 PM UTC+6, David Pilato wrote:
So you reindex into the same index and not in another clean one?
So you "update" docs, right?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 16:54, Kirill Teplinskiy tkiri...@gmail.com a écrit :

Thank you, David!

I need to call _optimize by hands to refresh payloads in completion suggester. This method is recommended here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/search-suggesters-completion.html. For some reason _suggest returns old payloads after reindexing on our stage server. I don't know why, completion suggester on my local instance of ElasticSearch updates on the fly. Maybe because our stage index is 10Gb size and contains 10 millions documents and my local index has only 180 000 documents.

On Thursday, May 29, 2014 7:37:24 PM UTC+6, David Pilato wrote:
In 99.9% you should not call optimize api and let elasticsearch/Lucene do it for you when needed.

To answer to your question, yes search and index operations will still possible during that time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2014 à 14:37, Kirill Teplinskiy tkiri...@gmail.com a écrit :

Hello!

Can anyone tell is it safe to call _optimize under normal production work? Will search requests be responded and indexing be performed?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5c75d64c-4fb6-4fe0-aaec-8dc3c5cf68f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/95b0bfcf-38b5-4880-ae50-efaa11aaddb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f830d7a1-8ad8-4390-9860-dc04029c7cd8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2d0e601e-55f6-456c-8901-42e074705d26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.5387591c.1e7ff521.28b%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(Sang Dang) #9

Hi David Pilato,
Do we have any plan to fix AutoCompletion when update output or payload ?
Currently, I also have problem when do "update" docs, I need to use
_optimize to force ES to rebuild FST.

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0e82b379-2e8d-4ab9-b124-06807848f92a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #10