Optimization Questions

Elastic Stack Elasticsearch
1

Hey Guys,
We were doing some updates to our es(1.3.1) clusters recently and had some
questions about _optimize. We optimized with max_num_segments 1 and we're
still seeing ~25 segments per shard. The index that was optimized had no
writes going to it during the time, it was actually freshly re-opened after
an upgrade. Also, are there any tricks to seeing when an optimize is done
other that watching merges stats and disk IO? Maybe some data in marvel?

Thanks for your assistance,
Greg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/17622c72-f004-4fda-92fb-dda393a64807%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi Greg,

I believe max_num_segments is technically a hint that can be overridden by the merge algorithm if it decides to. You might try simply re-running the optimize again to get from ~25 down closer to 1. Sorry but I don't know of any way to see when the optimize is finished - it's really just forcing a merge so looking at merge stats is what you want.

Hope that helps.
Andrew

On Aug 15, 2014, at 8:01 PM, Gregory Sutcliffe gsutcliffe@publishthis.com wrote:

Hey Guys,
We were doing some updates to our es(1.3.1) clusters recently and had some questions about _optimize. We optimized with max_num_segments 1 and we're still seeing ~25 segments per shard. The index that was optimized had no writes going to it during the time, it was actually freshly re-opened after an upgrade. Also, are there any tricks to seeing when an optimize is done other that watching merges stats and disk IO? Maybe some data in marvel?

Thanks for your assistance,
Greg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/17622c72-f004-4fda-92fb-dda393a64807%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/88238DE4-AFC4-41D0-B495-ED9938D7CB9C%40elasticsearch.com.
For more options, visit https://groups.google.com/d/optout.

3

You could turn on TRACE logging for the "lucene.iw" component. This will
give tons of details about what merges are being done.

Normally, if there are no writes going to the index at the same time, an
optimize with max_num_segments=1 really should get down to 1 segment in the
end ... not sure why it isn't in your case. Was there a refresh after the
optimize?

Mike McCandless

http://blog.mikemccandless.com

On Mon, Aug 18, 2014 at 12:33 PM, Andrew Selden <
andrew.selden@elasticsearch.com> wrote:

Hi Greg,

I believe max_num_segments is technically a hint that can be overridden by
the merge algorithm if it decides to. You might try simply re-running the
optimize again to get from ~25 down closer to 1. Sorry but I don’t know of
any way to see when the optimize is finished - it’s really just forcing a
merge so looking at merge stats is what you want.

Hope that helps.
Andrew

On Aug 15, 2014, at 8:01 PM, Gregory Sutcliffe gsutcliffe@publishthis.com
wrote:

Hey Guys,
We were doing some updates to our es(1.3.1) clusters recently and had some
questions about _optimize. We optimized with max_num_segments 1 and we're
still seeing ~25 segments per shard. The index that was optimized had no
writes going to it during the time, it was actually freshly re-opened after
an upgrade. Also, are there any tricks to seeing when an optimize is done
other that watching merges stats and disk IO? Maybe some data in marvel?

Thanks for your assistance,
Greg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/17622c72-f004-4fda-92fb-dda393a64807%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/17622c72-f004-4fda-92fb-dda393a64807%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/88238DE4-AFC4-41D0-B495-ED9938D7CB9C%40elasticsearch.com
https://groups.google.com/d/msgid/elasticsearch/88238DE4-AFC4-41D0-B495-ED9938D7CB9C%40elasticsearch.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRcMuZHzruXtto_K%3Dw7uNqABRzLe3rjSgd182iF6xBi5Gg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.