# Running \_forcemerge on an index that is being write on

**URL:** <https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492>\
**Category:** Elasticsearch\
**Created:** [February 25, 2023, 8:42am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492 "2023-02-25T08:42:25Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tudor\_Plugaru](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor_plugaru/32/117748_2.png) [@Tudor\_Plugaru](https://discuss.elastic.co/u/Tudor_Plugaru)\
**Post date:** [February 25, 2023, 8:42am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/1 "2023-02-25T08:42:25Z")

</div>

Hi,  
we are having an index with heavy updates on the documents. This leads us to having a lot of uncleaned deleted documents, for example, we can have around 400M searchable documents and around 150M of uncleaned documents. It's even worst on larger ES clusters that we are having. Since this ratio is pretty high, we want to look into running `_forcemerge` on the index without stopping writes on it, or maybe on a low traffic window, like weekens.  
What would be the impact on the index if we do this while keeping the writes on? I assume that having a bit less uncleaned documents, it will help ES manage segments faster and more reliable while also we can reclaim a bit of disk space.

You can see below another big cluster that we are having and the number of uncleaned documents.

 ![Screenshot 2023-02-25 at 10.40.27 AM](https://us1.discourse-cdn.com/elastic/original/3X/0/6/061a60369161e7131f6f935ea764746902e2953c.png)

---

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [February 25, 2023, 12:50pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/2 "2023-02-25T12:50:35Z")

</div>

I would not advise running a standard `_forcemerge` against an actively written index as it can cause issues, this is mentioned in the docs [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html#forcemerge-api-desc).

One option however if you really need to clean up the deleted docs for some reason is use `_forcemerge` with `only_expunge_deletes=true`. This will only remove segments that have a higher % of deleted docs. More about this query parameter can be found [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html#forcemerge-api-query-params).

Generally, I recommend just letting Elastic handle the cleanup process, but if you really need to do it manually for some reason, I'd recommend testing the command on a non-production index first to see if you have any negative performance impacts from it.

---

<div class="post-metadata">

**Author:** ![Tudor\_Plugaru](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor_plugaru/32/117748_2.png) [@Tudor\_Plugaru](https://discuss.elastic.co/u/Tudor_Plugaru)\
**Post date:** [February 25, 2023, 1:11pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/3 "2023-02-25T13:11:01Z")

</div>

Yes, I am aware that is not recommended to `_forcemerge` on an actively written index.

Thanks for the `only_expunge_deletes=True` tip tho, we might try that instead.

The main reason we want to try this is because of the high number of uncleaned documents. This is due to the nature of how we index the data as a lot of updates are made on the documents. We might let ES handle this, but then when there is a spike of updates on documents from a single shard, the merging kicks in and ES can't really handle the load of merging the segments and at the same time indexing documents on the same shard and we get degraded performance.  
So we are looking maybe to run `_forcemerge` from time to time, maybe during weekends to maybe ease the thing out. Not sure what else we can try...

---

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [February 25, 2023, 1:24pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/4 "2023-02-25T13:24:56Z")

</div>

> ES can't really handle the load of merging the segments and at the same time indexing documents on the same shard and we get degraded performance.

This is interesting. I'd say that `only_expunge_deletes=true` run off hours might help here, but another solution could be to split the index into more shards that can better distribute the load of the merges.

At this point I'd say the solution really depends more on _your_ needs though, as I think either would work.

---

<div class="post-metadata">

**Author:** ![Tudor\_Plugaru](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor_plugaru/32/117748_2.png) [@Tudor\_Plugaru](https://discuss.elastic.co/u/Tudor_Plugaru)\
**Post date:** [February 25, 2023, 1:35pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/5 "2023-02-25T13:35:32Z")

</div>

Yeah, it's a bit problematic for us to reindex at this point, but we are aware that we need to increase the shards number, it's just not a quick solution for us at the moment 😅 Hence us trying to find different other ways to fight this until we reindex everything.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 26, 2023, 8:12am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/6 "2023-02-26T08:12:13Z")

</div>

> [@Tudor\_Plugaru](#):
>
> we can have around 400M searchable documents and around 150M of uncleaned documents

I guess the main problem here is that Elasticsearch (well, Lucene) doesn't think this is a particularly high number of deleted docs. Merging is an expensive process, it's usually better to delay it until there are enough deleted docs to make it worthwhile.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 26, 2023, 8:17am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/7 "2023-02-26T08:17:44Z")

</div>

> [@Tudor\_Plugaru](#):
>
> ES can't really handle the load of merging the segments and at the same time indexing documents on the same shard and we get degraded performance

This suggests there might be something else wrong in your config. I'd recommend focussing on the reasons for this degraded performance directly. A period of high indexing load is going to lead to merges whether you force merge beforehand or not.

---

<div class="post-metadata">

**Author:** ![Tudor\_Plugaru](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor_plugaru/32/117748_2.png) [@Tudor\_Plugaru](https://discuss.elastic.co/u/Tudor_Plugaru)\
**Post date:** [February 26, 2023, 8:28am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/8 "2023-02-26T08:28:40Z")

</div>

Hi David,

yeah, I've actually run `_forcemerge` on the index while I stopped the indexing and it seems to worked out, disk space reclaimed which is nice.  
But we do seem to still have the problem with indexing since we are using `_bulk` api. I've tried everything from increasing/decreasing the bulk size, event with 10 update by script requests but it still very slow. From my observations, it seems that most of the updates are going to a single shard since we use `routing`. It constantly get's deallocated and allocated again. Not sure how to deal this this one to be honest.

Resource wise, we are using quite large machines, 16vCPU and 64GB of memory on Elastic Cloud hoping that more resources would help us with this, but it does not seem to be the case.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 26, 2023, 9:58am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/9 "2023-02-26T09:58:10Z")

</div>

> [@Tudor\_Plugaru](#):
>
> It constantly get's deallocated and allocated again.

Do you mean the shard is failing? If so, that definitely shouldn't happen, I recommend addressing this.

---

<div class="post-metadata">

**Author:** ![Tudor\_Plugaru](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor_plugaru/32/117748_2.png) [@Tudor\_Plugaru](https://discuss.elastic.co/u/Tudor_Plugaru)\
**Post date:** [February 26, 2023, 10:16am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/10 "2023-02-26T10:16:25Z")

</div>

Yes, this is what I am trying to figure out and I just take options one by one.  
The main problem being that most of the requests are going to a single shard which I assume ES is not able to handle it properly? Not sure what other options we have to speed the process, does disabling shard replicas would help? To only have primary shards and finish the indexing and then enable back replicas?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 26, 2023, 11:02am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/11 "2023-02-26T11:02:37Z")

</div>

It should not be possible to cause a shard to fail just by sending it too much indexing traffic. Instead, ES should just push back (typically by returning a `429 Too Many Requests`). So if it's not doing that for some reason then we need to understand why and address the fundamental problem. For instance, what is the exact nature of the shard failure? What appears in the ES logs, for instance, and in the responses to indexing requests? Perhaps this is a bug, but we cannot fix it without understanding it in more detail.

---

<div class="post-metadata">

**Author:** ![Tudor\_Plugaru](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor_plugaru/32/117748_2.png) [@Tudor\_Plugaru](https://discuss.elastic.co/u/Tudor_Plugaru)\
**Post date:** [February 26, 2023, 3:29pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/12 "2023-02-26T15:29:25Z")

</div>

In the end, we figured it out. It

It was because of the data. The documents we were trying to index were taking a lot of time to be indexed due to the nature of the data and because we run some regex expression at index time. Once we sorted that out, everything became normal again.

But on the topic of `_forcemerge`, I think we indeed have to run it for sure. The reason we are having that many uncleaned deleted documents is because we are having a lot of old big \>5GB segments with a lot of deleted documents.

```auto
index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version compound
index-name-v0 0 p ip-address-here _cwg 16720 2929550 2730209 5.2gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _d1y 16918 2866985 2800087 5.2gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _d6s 17092 3199872 2423136 5.4gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _dex 17385 3037376 2075589 5.4gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _dix 17529 3429513 2044662 5.2gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _dnc 17688 3464548 1821027 5.5gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _drz 17855 3223482 2061821 5.5gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _dx1 18037 3154251 2095340 5.4gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _e1u 18210 3478554 2015227 5.3gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _e88 18440 3621043 1688986 5.2gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _edw 18644 3670615 1842516 5.3gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _ely 18934 3883761 1323761 5.5gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _euw 19256 4342507 1078338 5.3gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _fbs 19864 4215320 885812 5.2gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _fwt 20621 4401008 865692 5.1gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _hbk4 808132 4390760 277667 4.8gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _v2d4 1449400 4845760 112970 5.1gb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _ved8 1464956 432762 240300 523.7mb 0 true true 9.3.0 true
index-name-v0 0 p ip-address-here _vfja 1466470 310035 43605 370.1mb 0 true true 9.3.0 true

```

And based on the [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html), normal merging does not run on segments larger than 5GB.

> But force merge can cause very large (\> 5GB) segments to be produced, which are not eligible for regular merges.

I find the working a bit confusing, but it seems to make sense in our case as based on the `generation` of the segments, we can see that we have very old segments that were always there...

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 26, 2023, 3:37pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/13 "2023-02-26T15:37:17Z")

</div>

> [@Tudor\_Plugaru](#):
>
> In the end, we figured it out. It

Great, but ...

> [@Tudor\_Plugaru](#):
>
> It was because of the data. The documents we were trying to index were taking a lot of time to be indexed due to the nature of the data and because we run some regex expression at index time. Once we sorted that out, everything became normal again.

... I still think this should not have led to shard failures, and would like to understand whether you were encountering a bug.

> [@Tudor\_Plugaru](#):
>
> we are having a lot of old big \>5GB segments with a lot of deleted documents.

Again, Elasticsearch doesn't think this is too many deleted documents. These deleted docs will be merged away when it becomes worth doing so.

---

<div class="post-metadata">

**Author:** ![Tudor\_Plugaru](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor_plugaru/32/117748_2.png) [@Tudor\_Plugaru](https://discuss.elastic.co/u/Tudor_Plugaru)\
**Post date:** [February 26, 2023, 3:53pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/14 "2023-02-26T15:53:06Z")

</div>

> [@DavidTurner](#):
>
> I still think this should not have led to shard failures, and would like to understand whether you were encountering a bug.

Well, I am ready to help digging into the issue, I just don't know how :). I can provide a deployment id if you are interested as we run on Elastic Cloud, or anything else if it's needed. Just DM me 🙂

> Again, Elasticsearch doesn't think this is too many deleted documents. These deleted docs will be merged away when it becomes worth doing so.

But then how come those segments I showed above were not merged? The ones larger than 5GB. We have created the index around 4 months ago and since then, we are constantly writing to it using `_bulk` API.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 26, 2023, 5:38pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/15 "2023-02-26T17:38:46Z")

</div>

> [@Tudor\_Plugaru](#):
>
> But then how come those segments I showed above were not merged?

They don't have enough deleted docs to be a merge candidate yet: the live docs are still more than 2.5GiB in all cases so merging any of them together would yield a segment that's even larger than 5GiB, which is something Lucene deliberately avoids. Once there's a way to merge them together without (significantly) exceeding the 5GiB target size, Elasticsearch will do so.

---

<div class="post-metadata">

**Author:** ![Tudor\_Plugaru](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor_plugaru/32/117748_2.png) [@Tudor\_Plugaru](https://discuss.elastic.co/u/Tudor_Plugaru)\
**Post date:** [February 26, 2023, 6:01pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/16 "2023-02-26T18:01:14Z")

</div>

> [@DavidTurner](#):
>
> They don't have enough deleted docs to be a merge candidate yet:

I see. That is interesting, but then there is the cost efficiency problem, no? In our case, on the deployment I ran `_forcemerge` and shared with you the id, we reclaimed almost 1Tb of disk space just by cleaning up the deleted documents. In our case this is important as we mostly scale up the clusters due to running out of disk space.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 26, 2023, 6:26pm UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/17 "2023-02-26T18:26:18Z")

</div>

The shard failures do indeed look to be a bug: somehow your traffic pattern is generating messages between nodes which are so large that the receiving node briefly leaves the cluster. I don't really expect Elasticsearch to handle such large messages, but it should fail more gracefully than it does currently (and ideally should not fail the shard in this situation). I opened [Transport messages exceeding 2GiB are not handled gracefully · Issue #94137 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/94137).

> [@Tudor\_Plugaru](#):
>
> That is interesting, but then there is the cost efficiency problem, no?

I do see your point, maybe we could find some way to reclaim this disk space more enthusiastically in cases where disk space is the limiting factor and savings are available. Please open an issue on Github to start a discussion about this.

---

<div class="post-metadata">

**Author:** ![Tudor\_Plugaru](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor_plugaru/32/117748_2.png) [@Tudor\_Plugaru](https://discuss.elastic.co/u/Tudor_Plugaru)\
**Post date:** [February 27, 2023, 7:52am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/18 "2023-02-27T07:52:16Z")

</div>

> [@DavidTurner](#):
>
> somehow your traffic pattern is generating messages between nodes which are so large that the receiving node briefly leaves the cluster.

Oh, that is very interesting and useful. Right now we sometimes have quite large bulk sizes both in terms of items or size. Should we lower them then? Also, how can we monitor this?

> I do see your point, maybe we could find some way to reclaim this disk space more enthusiastically in cases where disk space is the limiting factor and savings are available. Please open an issue on Github to start a discussion about this.

I'll do that. Thanks

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 27, 2023, 8:22am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/19 "2023-02-27T08:22:27Z")

</div>

> [@Tudor\_Plugaru](#):
>
> Right now we sometimes have quite large bulk sizes both in terms of items or size. Should we lower them then?

Bulk requests are limited in size to 100MiB at the HTTP layer, so for the request to reach 2GiB you must be adding the vast majority of the data (≥95%) as the request passes through your ingest pipelines. Smaller bulks would help keep the final size under 2GiB, but also maybe your ingest pipelines could be simplified?

> [@Tudor\_Plugaru](#):
>
> Also, how can we monitor this?

I don't know of a good way to see the post-pipeline bulk size unfortunately. I'm not sure anyone has hit this problem before. I couldn't find any other clusters in all of Elastic Cloud that had experienced anything similar.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 27, 2023, 8:23am UTC](https://discuss.elastic.co/t/running-forcemerge-on-an-index-that-is-being-write-on/326492/20 "2023-03-27T08:23:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.