Experimentally, I executed index optimization API but I am not seeing any
merging activities in elasticsearch side. I am watching the number of index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>, I don't see any changes.
How can I know what is happening for optimization in elasticsearch?
Hm, I can't think of a direct way ES exposes this. There may be some way,
but I can't think of it right now. However, one thing you could look at is
disk IO. If optimization is happening, you should see more disk IO if you
look at your system monitoring tool or run iostats, vmstat, etc.
It's also possible that your index is optimized already, so there is
nothing to optimize.
On Tuesday, November 6, 2012 8:40:12 PM UTC-5, Jae wrote:
Experimentally, I executed index optimization API but I am not seeing any
merging activities in elasticsearch side. I am watching the number of index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>, I don't see any changes.
How can I know what is happening for optimization in elasticsearch?
Besides monitoring your IO, you can check out your merges through the Stats API:
Please note that you would need to explicitly add merge=true as a
parameter to get the merge status.
Also, you might want to check out the number of segments before and
after optimization through the Indices Segments API:
AFAIK the Optimize API by default only does a merge if it's needed
according to your Merge Policy:
And normally, this is done automatically in background, so in a lot
situations just calling the Optimize API wouldn't do anything. You
would probably want to specify a rather small number_of_segments value
there, like 5. But that would be pretty heavy on IO, so you might want
to schedule these optimizations for off-peak hours.
Experimentally, I executed index optimization API but I am not seeing any
merging activities in elasticsearch side. I am watching the number of index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>,
I don't see any changes.
How can I know what is happening for optimization in elasticsearch?
And normally, this is done automatically in background, so in a lot
situations just calling the Optimize API wouldn't do anything. You
would probably want to specify a rather small number_of_segments value
there, like 5. But that would be pretty heavy on IO, so you might want
to schedule these optimizations for off-peak hours.
Experimentally, I executed index optimization API but I am not seeing any
merging activities in elasticsearch side. I am watching the number of index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>,
I don't see any changes.
How can I know what is happening for optimization in elasticsearch?
Optimizing isn't really necessary unless you're worried about file handles
or if your data is fairly static. Having multiple segments is not going to
cause any problems. I would recommend just letting merging occur.
On Thursday, November 8, 2012 4:22:06 AM UTC+11, Jae wrote:
Thank you so much for your answers.
My bad, I thought the default value of max_num_segments was 1 but it
wasn't. I didn't specify max_num_segments as 1.
Do you recommend calling optimize API with max_num_segments as 1 or
leave it to the default merge function?
I am afraid that full optimization can cause something erroneous such as
OOM.
Best, Jae
On Wed, Nov 7, 2012 at 1:00 AM, Radu Gheorghe
<radu.g...@sematext.com <javascript:>> wrote:
Hello Jae,
Besides monitoring your IO, you can check out your merges through the
Stats API:
And normally, this is done automatically in background, so in a lot
situations just calling the Optimize API wouldn't do anything. You
would probably want to specify a rather small number_of_segments value
there, like 5. But that would be pretty heavy on IO, so you might want
to schedule these optimizations for off-peak hours.
On Wed, Nov 7, 2012 at 3:40 AM, Jae <meta...@gmail.com <javascript:>>
wrote:
Experimentally, I executed index optimization API but I am not seeing
any
merging activities in elasticsearch side. I am watching the number of
index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>,
I don't see any changes.
How can I know what is happening for optimization in elasticsearch?
optimizing is really a heavy operation and should only be executed if
really necessary. if you are doing batch indexing and your index doesn't
change frequently then optimizing makes sense. If you do get frequent
updates you will be "un-optimized" at the very moment you execute the
command. for more details read my blog about optimizing: http://www.searchworkings.org/blog/-/blogs/simon-says%3A-optimize-is-bad-for-you
simon
On Wednesday, November 7, 2012 6:22:06 PM UTC+1, Jae wrote:
Thank you so much for your answers.
My bad, I thought the default value of max_num_segments was 1 but it
wasn't. I didn't specify max_num_segments as 1.
Do you recommend calling optimize API with max_num_segments as 1 or
leave it to the default merge function?
I am afraid that full optimization can cause something erroneous such as
OOM.
Best, Jae
On Wed, Nov 7, 2012 at 1:00 AM, Radu Gheorghe
<radu.g...@sematext.com <javascript:>> wrote:
Hello Jae,
Besides monitoring your IO, you can check out your merges through the
Stats API:
And normally, this is done automatically in background, so in a lot
situations just calling the Optimize API wouldn't do anything. You
would probably want to specify a rather small number_of_segments value
there, like 5. But that would be pretty heavy on IO, so you might want
to schedule these optimizations for off-peak hours.
On Wed, Nov 7, 2012 at 3:40 AM, Jae <meta...@gmail.com <javascript:>>
wrote:
Experimentally, I executed index optimization API but I am not seeing
any
merging activities in elasticsearch side. I am watching the number of
index
files in /<CLUSTER_NAME>/nodex/0/indices//<SHARD_ID>,
I don't see any changes.
How can I know what is happening for optimization in elasticsearch?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.