Elastic Search max_num_segments in forcemerge

Elastic Stack Elasticsearch
1

Hi,

I am currently using ES v6.1.2. I have an index of size 1.3 tb consisting of 16 shards (8 primaries).
The primary shard size amounts to around 700GB.

I am planning to perform a forcemerge operation on the index.
Any suggestions, what should be the max_num_segment value. As far as I know, by default, it takes the value as 1.

I want my search speed to improve. So what should be the ideal value for max_num_segments.

Thanks in advance :slight_smile:

2

Hi,

Any update?
Kindly help. Need to cater to this on an urgent basis.

Regards,
Raj Shah

3

1 is ok.
But more than that you can probably do everything with fewer shards. Try with 1 shard only.
The shrink API might help.

4

How can the size of the primary shards be 700MB if the total size of the index across 16 shards is 1.3TB?

5

Hey @Christian_Dahlqvist,

I am sorry. That should read 700GB.

Regards,
Raj Shah

6

That invalidates my answer.

7

Hey @dadoonet,

Sorry for the typing error before.
Just to confirm, should I go ahead with max_seg_count as 1?
Or we should start with some higher number for this volume of data?

Regards,
Raj Shah

8

As long as you are not writing anymore to this index, I think that could be ok to set it to 1.
@Christian_Dahlqvist WDYT?

9

Hi @dadoonet,

We do perform write operation once everyday.

Regards,
Raj Shah

10

Do you batch update the index once a day or is it a continuous update?

11

Hi @Christian_Dahlqvist,

It happens only once a day.

12

Performing a forcemerge on shards that large will use up a lot of resources. I would recommend trying it once and see if it brings the benefits you are hoping for. I think it should be OK to set max)num_segments to 1, so that is what I would try with.

13

Hi @Christian_Dahlqvist & @dadoonet ,

Thanks for your prompt response.
Apart from the process being resource intensive, I would like to know if it has any other disadvantages.
Since the data set is quite large, just want to be sure about it.

14

It can end up taking a long time and use a lot of disk I/O, so could affect users while it is progressing. For that reason I would recommend trying it out in a test environment rather that trying it in production.

15

Hello,

Thank you for the help. I completed the force_merge operation on 1.3 TB index.
The segment count is now 1 per shard.
It took 3.5 hours to complete force_merge process.

We did force_merge 1 year after creating the index. Now, if we plan to do force_merge every 6 months, will it take the same amount of time or it should be lesser?

Thanks in advance :slight_smile:

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.