Index optimization behavior

Elastic Stack Elasticsearch
#1

Experimentally, I executed index optimization API but I am not seeing any
merging activities in elasticsearch side. I am watching the number of index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>, I don't see any changes.

How can I know what is happening for optimization in elasticsearch?

Thank you
Best, Jae

--

#2

Hi,

Hm, I can't think of a direct way ES exposes this. There may be some way,
but I can't think of it right now. However, one thing you could look at is
disk IO. If optimization is happening, you should see more disk IO if you
look at your system monitoring tool or run iostats, vmstat, etc.

It's also possible that your index is optimized already, so there is
nothing to optimize.

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html

On Tuesday, November 6, 2012 8:40:12 PM UTC-5, Jae wrote:

Experimentally, I executed index optimization API but I am not seeing any
merging activities in elasticsearch side. I am watching the number of index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>, I don't see any changes.

How can I know what is happening for optimization in elasticsearch?

Thank you
Best, Jae

--

#3

Hello Jae,

Besides monitoring your IO, you can check out your merges through the Stats API:
http://www.elasticsearch.org/guide/reference/api/admin-indices-stats.html

Please note that you would need to explicitly add merge=true as a
parameter to get the merge status.

Also, you might want to check out the number of segments before and
after optimization through the Indices Segments API:
http://www.elasticsearch.org/guide/reference/api/admin-indices-segments.html

AFAIK the Optimize API by default only does a merge if it's needed
according to your Merge Policy:
http://www.elasticsearch.org/guide/reference/index-modules/merge.html

And normally, this is done automatically in background, so in a lot
situations just calling the Optimize API wouldn't do anything. You
would probably want to specify a rather small number_of_segments value
there, like 5. But that would be pretty heavy on IO, so you might want
to schedule these optimizations for off-peak hours.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Wed, Nov 7, 2012 at 3:40 AM, Jae metacret@gmail.com wrote:

Experimentally, I executed index optimization API but I am not seeing any
merging activities in elasticsearch side. I am watching the number of index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>,
I don't see any changes.

How can I know what is happening for optimization in elasticsearch?

Thank you
Best, Jae

--

--

#4

Thank you so much for your answers.

My bad, I thought the default value of max_num_segments was 1 but it
wasn't. I didn't specify max_num_segments as 1.

Do you recommend calling optimize API with max_num_segments as 1 or
leave it to the default merge function?

I am afraid that full optimization can cause something erroneous such as OOM.

Best, Jae

On Wed, Nov 7, 2012 at 1:00 AM, Radu Gheorghe
radu.gheorghe@sematext.com wrote:

Hello Jae,

Besides monitoring your IO, you can check out your merges through the Stats API:
http://www.elasticsearch.org/guide/reference/api/admin-indices-stats.html

Please note that you would need to explicitly add merge=true as a
parameter to get the merge status.

Also, you might want to check out the number of segments before and
after optimization through the Indices Segments API:
http://www.elasticsearch.org/guide/reference/api/admin-indices-segments.html

AFAIK the Optimize API by default only does a merge if it's needed
according to your Merge Policy:
http://www.elasticsearch.org/guide/reference/index-modules/merge.html

And normally, this is done automatically in background, so in a lot
situations just calling the Optimize API wouldn't do anything. You
would probably want to specify a rather small number_of_segments value
there, like 5. But that would be pretty heavy on IO, so you might want
to schedule these optimizations for off-peak hours.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Wed, Nov 7, 2012 at 3:40 AM, Jae metacret@gmail.com wrote:

Experimentally, I executed index optimization API but I am not seeing any
merging activities in elasticsearch side. I am watching the number of index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>,
I don't see any changes.

How can I know what is happening for optimization in elasticsearch?

Thank you
Best, Jae

--

--

--

#5

Jae,

Optimizing isn't really necessary unless you're worried about file handles
or if your data is fairly static. Having multiple segments is not going to
cause any problems. I would recommend just letting merging occur.

On Thursday, November 8, 2012 4:22:06 AM UTC+11, Jae wrote:

Thank you so much for your answers.

My bad, I thought the default value of max_num_segments was 1 but it
wasn't. I didn't specify max_num_segments as 1.

Do you recommend calling optimize API with max_num_segments as 1 or
leave it to the default merge function?

I am afraid that full optimization can cause something erroneous such as
OOM.

Best, Jae

On Wed, Nov 7, 2012 at 1:00 AM, Radu Gheorghe
<radu.g...@sematext.com <javascript:>> wrote:

Hello Jae,

Besides monitoring your IO, you can check out your merges through the
Stats API:

http://www.elasticsearch.org/guide/reference/api/admin-indices-stats.html

Please note that you would need to explicitly add merge=true as a
parameter to get the merge status.

Also, you might want to check out the number of segments before and
after optimization through the Indices Segments API:

http://www.elasticsearch.org/guide/reference/api/admin-indices-segments.html

AFAIK the Optimize API by default only does a merge if it's needed
according to your Merge Policy:
http://www.elasticsearch.org/guide/reference/index-modules/merge.html

And normally, this is done automatically in background, so in a lot
situations just calling the Optimize API wouldn't do anything. You
would probably want to specify a rather small number_of_segments value
there, like 5. But that would be pretty heavy on IO, so you might want
to schedule these optimizations for off-peak hours.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Wed, Nov 7, 2012 at 3:40 AM, Jae <meta...@gmail.com <javascript:>>
wrote:

Experimentally, I executed index optimization API but I am not seeing
any

merging activities in elasticsearch side. I am watching the number of
index

files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>,

I don't see any changes.

How can I know what is happening for optimization in elasticsearch?

Thank you
Best, Jae

--

--

--

#6

hey Jae,

optimizing is really a heavy operation and should only be executed if
really necessary. if you are doing batch indexing and your index doesn't
change frequently then optimizing makes sense. If you do get frequent
updates you will be "un-optimized" at the very moment you execute the
command. for more details read my blog about optimizing:
http://www.searchworkings.org/blog/-/blogs/simon-says%3A-optimize-is-bad-for-you

simon

On Wednesday, November 7, 2012 6:22:06 PM UTC+1, Jae wrote:

Thank you so much for your answers.

My bad, I thought the default value of max_num_segments was 1 but it
wasn't. I didn't specify max_num_segments as 1.

Do you recommend calling optimize API with max_num_segments as 1 or
leave it to the default merge function?

I am afraid that full optimization can cause something erroneous such as
OOM.

Best, Jae

On Wed, Nov 7, 2012 at 1:00 AM, Radu Gheorghe
<radu.g...@sematext.com <javascript:>> wrote:

Hello Jae,

Besides monitoring your IO, you can check out your merges through the
Stats API:

http://www.elasticsearch.org/guide/reference/api/admin-indices-stats.html

Please note that you would need to explicitly add merge=true as a
parameter to get the merge status.

Also, you might want to check out the number of segments before and
after optimization through the Indices Segments API:

http://www.elasticsearch.org/guide/reference/api/admin-indices-segments.html

AFAIK the Optimize API by default only does a merge if it's needed
according to your Merge Policy:
http://www.elasticsearch.org/guide/reference/index-modules/merge.html

And normally, this is done automatically in background, so in a lot
situations just calling the Optimize API wouldn't do anything. You
would probably want to specify a rather small number_of_segments value
there, like 5. But that would be pretty heavy on IO, so you might want
to schedule these optimizations for off-peak hours.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Wed, Nov 7, 2012 at 3:40 AM, Jae <meta...@gmail.com <javascript:>>
wrote:

Experimentally, I executed index optimization API but I am not seeing
any

merging activities in elasticsearch side. I am watching the number of
index

files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>,

I don't see any changes.

How can I know what is happening for optimization in elasticsearch?

Thank you
Best, Jae

--

--

--

#7