# Can I put or update document when calling 'force merge' an index? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 4, 2018, 8:34am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039 "2018-01-04T08:34:25Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![michaelcheung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaelcheung/32/53285_2.png) [@michaelcheung](https://discuss.elastic.co/u/michaelcheung)\ **Post date:** [January 4, 2018, 8:34am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/1 "2018-01-04T08:34:25Z")
As mentioned in topic, I call force merge api, to manually clean up deleted docs in an index. Is it possible and safe to update document or put new document in 'merging' index? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 4, 2018, 2:05pm UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/2 "2018-01-04T14:05:48Z")
> [@michaelcheung](#): > > Is it possible and safe to update document or put new document in 'merging' index? Yes but we generally advise reserving the pain and cost of performing "force merge" operations on indices where you don't expect to receive any more updates. Automatic merging happens naturally in the background as new docs are added or updated. Your "helping hand" is not typically required with merges unless you know something elasticsearch doesn't eg you know there's no more content to be added to an index. ---
**Author:** ![michaelcheung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaelcheung/32/53285_2.png) [@michaelcheung](https://discuss.elastic.co/u/michaelcheung)\ **Post date:** [January 5, 2018, 3:57am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/3 "2018-01-05T03:57:01Z")
Thanks @Mark_Harwood. I want to further ask a question. What is the condition of automatic merging? (perhaps I have to study lucene's mechanism?) I have this concern because my index has 6.7m docs.count and 6.5m docs.deleted, that I hope those deleted docs can be cleaned up to free up storage. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 5, 2018, 10:08am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/4 "2018-01-05T10:08:22Z")
> [@michaelcheung](#): > > What is the condition of automatic merging? (perhaps I have to study lucene's mechanism?) I Mike McCandless has a great [blog post](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html) on the Lucene internals. > [@michaelcheung](#): > > my index has 6.7m docs.count and 6.5m docs.deleted, Ordinarily segments with \>50% are targets for rewriting through merge operations, My guess is you've previously called force merge to create a single large segment and then done a bunch of deletes. This is where things get tricky because there's another limit in play which is the maximum segment size Lucene is allowed to create. Ordinarily, "natural" writes and merges maintain segments below this threshold but a force merge operation can create an uber-segment which exceeds this size which essentially becomes unmanaged at this point. Despite having \> 50% deletes it is seen as too large and costly to be rewritten due to the max seg size threshold having been exceeded. This serves as a good case in point for user intervention conflicting with objectives of internal algorithms and this is why we've progressively tried to remove user-facing controls over merge operations - it's like allowing 2 people to try drive the same car. It's worth asking if your deletes are part of a data retention policy that ages out content e.g. deleting all data from the previous month? If this is the case you should look at using multiple [time based indices](https://www.elastic.co/guide/en/elasticsearch/guide/current/time-based.html) rather than maintaining a single index. It is cheaper and quicker to drop a whole index rather than delete individual docs from an existing index. ---
**Author:** ![michaelcheung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaelcheung/32/53285_2.png) [@michaelcheung](https://discuss.elastic.co/u/michaelcheung)\ **Post date:** [January 6, 2018, 4:40am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/5 "2018-01-06T04:40:53Z")
I have a number of document update operations on that index. As I understand, calling update actually delete current document and create a new one with updated values. So frequent update results to my current issue. In my production ES, I do have time-based indices, which are deleted directly once 'outdated'. But some important data is designated to a 'permanent' index. For example, to keep purchase status of every customer, my approach is to update 'last\_purchase\_date' and 'total\_purchase' fields on doc\_id: customerA. I cannot put these data to time-base indices as I have to keep full list of all customers, and increment total purchase value based on old document. It would be nice if there are better methods for my cases. Thanks. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 6, 2018, 5:38pm UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/6 "2018-01-06T17:38:54Z")
Ok. So 6.7m docs minus 6.5m deletes = 0.2m customers? As I mentioned before it sounds like the use of "force\_merge" has left you with an oversized segment that is not being tidied automatically. ---
**Author:** ![michaelcheung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaelcheung/32/53285_2.png) [@michaelcheung](https://discuss.elastic.co/u/michaelcheung)\ **Post date:** [January 8, 2018, 11:19am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/7 "2018-01-08T11:19:44Z")
Customer is one case. There are other types of documents using same update approach to store permanent data. As I remember, I did not force merge the index. But I tried to reindex from index A to index B, that index A was an old index force merged before and index B is my 6.7m-docs index. I guess it is unlikely the reason. Or any related metrics/settings I can provide to look into the issue? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 8, 2018, 11:29am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/8 "2018-01-08T11:29:28Z")
> [@michaelcheung](#): > > Or any related metrics/settings I can provide to look into the issue? We can deep-dive into the segments in your index using: ``` http://localhost:9200/YOUR_INDEX_NAME/_segments?pretty ``` ---
**Author:** ![michaelcheung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaelcheung/32/53285_2.png) [@michaelcheung](https://discuss.elastic.co/u/michaelcheung)\ **Post date:** [January 8, 2018, 11:41am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/9 "2018-01-08T11:41:16Z")
Please check the link for the outputs. It exceeds word limit so I have to use google drive. > **[segments.txt](https://docs.google.com/document/d/1hlKhwXfuxt0lPfByI5bMCgJaBNOMD_MIwIsZmq_Im6M/edit?usp=sharing)** > > { "\_shards": { "total": 1, "successful": 1, "failed": 0 }, "indices": { "xxx": { "shards": { "0": \[ { "routing": { "state": "STARTED", "primary": true, ... ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 8, 2018, 12:33pm UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/10 "2018-01-08T12:33:11Z")
My mistake. I assumed you had 6.7m docs of which 6.5m docs were deleted and this was mostly in one segment. Things aren't nearly as bad as I thought and behaving normally. You currently have 37% of your space given to deleted docs likely caused by frequent updates . All segments are hovering under the 50% deleted threshold which will trigger their merging. Here are some breakdowns of your results where each bar is a segment: ![Segments_-_Google_Sheets](https://us1.discourse-cdn.com/elastic/original/3X/8/5/85432ea16bfec0d8f89dc567acd7bfc5f9311093.png) ---
**Author:** ![michaelcheung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaelcheung/32/53285_2.png) [@michaelcheung](https://discuss.elastic.co/u/michaelcheung)\ **Post date:** [January 15, 2018, 9:12am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/11 "2018-01-15T09:12:34Z")
Sorry for a late reply. I found that i interpreted wrongly about cat indices api in ES. The columns of docs.count and docs.deleted are mutually exclusive. Thank you for your illustration showing the index is normal. I also see value drop in deleted docs last week. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 12, 2018, 9:12am UTC](https://discuss.elastic.co/t/can-i-put-or-update-document-when-calling-force-merge-an-index/114039/12 "2018-02-12T09:12:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.