# \_forcemerge on ES 7.3 issue
**URL:** https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571
**Category:** Elasticsearch
**Created:** [August 30, 2019, 6:57pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571 "2019-08-30T18:57:12Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@jeroen1](https://discuss.elastic.co/u/jeroen1)
#### Post date: [August 30, 2019, 6:57pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/1 "2019-08-30T18:57:12Z")
I've upgraded hardware. Clean install of ES 7.3. Importing the same static dataset I was using on my old ES 6 box. Importing on 7 is faster BUT... after completing the import I'm running a \_forcemerge on the ~1.1TB dataset. On 6 this runs for some time, slowly but surely decreasing segments count and disk space used. On 7 there're an issue: the same command on the same dataset already runs for quite some time and the segment count is slowly INCREASING and disk usages of the index already more than doubled (from 1.1TB to \>2.2 TB).
Any clues about what's going on and why ES / Lucene is using \>100% temp storage? Is there a workaround to not use all this temp storage?
---
### Author:  [@jeroen1](https://discuss.elastic.co/u/jeroen1)
#### Post date: [September 1, 2019, 8:01pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/2 "2019-09-01T20:01:39Z")
This is what is looks like:
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 1, 2019, 9:31pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/3 "2019-09-01T21:31:44Z")
Are you still indexing into this index while the force-merge is running, or did you import everything, wait for that to complete, and _then_ request a force-merge? What exactly was the request you made to the force-merge API? Was the cluster health green throughout?
---
### Author:  [@jeroen1](https://discuss.elastic.co/u/jeroen1)
#### Post date: [September 1, 2019, 10:26pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/4 "2019-09-01T22:26:02Z")
> Are you still indexing into this index while the force-merge is running,
No.
> or did you import everything, wait for that to complete, and _then_ request a force-merge?
Yes.
> What exactly was the request you made to the force-merge API?
curl -H'Content-Type: application/json' -XPOST 'localhost:9200/myindex/\_forcemerge?max\_num\_segments=1&pretty'
> Was the cluster health green throughout?
Yes.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 2, 2019, 7:12am UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/5 "2019-09-02T07:12:51Z")
Does the disk space reduce again if you flush and refresh on this index?
---
### Author:  [@jeroen1](https://discuss.elastic.co/u/jeroen1)
#### Post date: [September 2, 2019, 8:55am UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/6 "2019-09-02T08:55:26Z")
> Does the disk space reduce again if you flush and refresh on this index?
What commands would you recommend to flush and refresh? Thanks.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 2, 2019, 9:08am UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/7 "2019-09-02T09:08:55Z")
This should be fine:
```auto
POST /myindex/_flush
POST /myindex/_refresh
```
---
### Author:  [@jeroen1](https://discuss.elastic.co/u/jeroen1)
#### Post date: [September 2, 2019, 3:24pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/8 "2019-09-02T15:24:39Z")
Update: merge process continued until 100% of disk space was taken. At that time the merge took \> 2x index size(!). The \_forcemerge however seems to be successful ("failed" : 0). Ran \_flush and \_refresh, waited a few minutes. Result: number of segments in the end went DOWN to a value lower than requested (I used rather high value instead of 1 to see if it would work anyway), disk space back to normal. Hmmm, that's pretty strange... Perhaps the process was already finished before but kept on writing data for a magic reason? Is there a way I can provide you with debug information?
Something that caught my eye in the storage directory: I see large files with both namens like _Lucene50_ and _Lucene80_. Seems like v5 and v8 files for an amateur like me, where I expect v8 only. Is this correct? The system contains a clean ES 7.3 install, clean generation of the index (no upgrades / re-indexing from older versions). Other info: using best\_compression and large ngram tokenizers.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 2, 2019, 5:21pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/9 "2019-09-02T17:21:59Z")
Thanks for the update. It is possible that this is related to [https://github.com/elastic/elasticsearch/pull/46066](https://github.com/elastic/elasticsearch/pull/46066) in which Elasticsearch isn't always as enthusiastic about flushing as perhaps it should be. Did you try flushing while the force-merge was ongoing too, or only at the end? Can you wait for the release of 7.4.0 and then try again?
> [@jeroen1](#):
>
> I see large files with both namens like **Lucene50** and **Lucene80**
I will check, but I don't think this is something to worry about.
---
### Author:  [@jeroen1](https://discuss.elastic.co/u/jeroen1)
#### Post date: [September 2, 2019, 5:42pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/10 "2019-09-02T17:42:20Z")
> It is possible that this is related to [https://github.com/elastic/elasticsearch/pull/46066](https://github.com/elastic/elasticsearch/pull/46066) in which Elasticsearch isn't always as enthusiastic about flushing as perhaps it should be.
Sounds plausible, thanks for the link!
> Did you try flushing while the force-merge was ongoing too, or only at the end?
Only at the end. Will try next time when ongoing.
> Can you wait for the release of 7.4.0 and then try again?
Yes, it's a non-production box used to make sure that there a no similar issues in production after a 'quick' upgrade 😉
What would you suggest for now, e.g. screening the following during the merge?
```
while true
do
sleep 600
curl -H'Content-Type: application/json' -XPOST 'localhost:9200/myindex/_flush
done
```
Or something more sophisticated?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 2, 2019, 6:12pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/11 "2019-09-02T18:12:02Z")
> [@DavidTurner](#):
>
> I will check, but I don't think this is something to worry about.
Confirmed, this is fine.
> [@jeroen1](#):
>
> What would you suggest for now, e.g. screening the following during the merge?
If that works then it seems like a reasonable workaround. We'd like to know if it does work, because if it doesn't then you might have hit a different (and as-yet-unknown) issue instead.
---
### Author:  [@jeroen1](https://discuss.elastic.co/u/jeroen1)
#### Post date: [September 3, 2019, 7:37pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/12 "2019-09-03T19:37:00Z")
> If that works then it seems like a reasonable workaround. We'd like to know if it does work, because if it doesn't then you might have hit a different (and as-yet-unknown) issue instead.
I've started a new run, using a smaller dataset, for faster results. Similar behavior: segment count slowly grows (difficult to see on the screendump because of the huge drop, but still the case) and disk usage significantly increased over the hours. Then your suggested workaround (flush and refresh), and...
Looking good 🤟
So it seems to be the same or a similar issues as described @ [https://github.com/elastic/elasticsearch/pull/46066](https://github.com/elastic/elasticsearch/pull/46066).
Conclusion: suggested workaround is effective in 7.3, structural fix as soon as 7.4 is released right?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 3, 2019, 8:46pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/13 "2019-09-03T20:46:57Z")
Great, thanks for reporting back. I also expect 7.4 contains a fix for this, yes, but please let us know if the problem isn't fixed there and we'll investigate further.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 1, 2019, 8:47pm UTC](https://discuss.elastic.co/t/forcemerge-on-es-7-3-issue/197571/14 "2019-10-01T20:47:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.