Disk space, delete-by-query, forcemerge

Elastic Stack Elasticsearch
1

Hi,

I have an index with ~ 100k doc.
Every 5 minutes, I add in this index ~ 50k docs and delete the same number.

This is not that much, however, each time, the disk space increase by ~ 15Mo, so after 12 hours the disk space is 2Go !

I ready many posts, the more relevants being this one or that one, however the _forcemerge +max_num_segments request seems to have no effect on the diskspace, not a single Mo released ...

I have the same pb on different ES versions: 6.3.0, 6.3.2, and 6.4

And in different environments (staging: only one node. production: one cluster, 3 nodes, each index on 5 shards + 1 rep)

Any hint or explanation would be great :slight_smile:

++

2

Hey,

if you stop writing in abbreviations it would be much easier to rd (bad pun intended).

can you share the output of GET yourindex/_stats here to check how many documents are marked as deleted? Can you share the output before and after running forcemerge?

Also, can you explain what the issue is with needing more space? What are you afraid of? running out of disk space?

Also the forcemerge docs contain a statement to only run against read only indices, which is not the case here.

One last hint, there are a few settings you can tune in order to have more aggressive merging going to due to the number of deletes in a segment, but they are not exposed in the documentation as they are considered expert settings. You can find them here

--Alex

3

Hi Alex,

Thank you for your help, and sorry for the abbreviations, I will try to improve my habits :slight_smile:

Here are the output of _stats on my index:

On the disk, this index takes 2.1 Go (and 18 hours ago, the same index was taking only ~ 10Mo !)

And yes I am afraid of running out of disk space, that's what is happening every day if I do not a full reset of the index (I delete the index, then recreate it, but that won't be possible once the site in production).

Thanks also for the forcemerge advanced options, I will read and try some of them.

++

4

Almost all your disk space seems to be taken up by the transaction log:

"translog": {
  "operations": 5247477,
  "size_in_bytes": 2221755825,
  "uncommitted_operations": 5247477,
  "uncommitted_size_in_bytes": 2221755825,
  "earliest_last_modified_age": 0
},

Make sure you have the refresh interval set to a actual duration in your index settings. It can somethings be useful to set this to -1 in order to disable it during heavy indexing, but you then need to reset it afterwards (or run a manual refresh). In newer versions the transaction logs is also kept for a while in order to make recovery more efficient. You can reduce the duration and size using these parameters, which may also reduce disk usage.

5

Indeed, you are right, I did not notice that this was the transaction log that took so much disp space.
I will try to play with the refresh interval plus the translog parameters and tell you if it works
thanks !

6

I tried these 2 commands:

POST /my-index/_refresh

POST /my-index/_flush?wait_if_ongoing=true&force=true

But that did not decrease the disk space.

After the flush, my /my-index/_stats looked like that:

"translog": {
    "operations": 2138851,
    "size_in_bytes":  905299183,
    "uncommitted_operations":  0,
    "uncommitted_size_in_bytes": 275,
    "earliest_last_modified_age": 0
}

And I noticed the "uncommited_size_in_bytes" which is not null, I was wondering if that could be the reason why the translog is not cleaned ?

However, in the translog documentation you pointed to me, I found a solution, by setting index.translog.retention.age to 1 hour, my folder size has decreased from 1Go to 100Mo !

So the problem seems solved :slight_smile:

Thanks for your help !

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.