Total store size of index with only one shard and one replica is three times the size of primary

Elastic Stack Elasticsearch
1

53
Screenshot 2019-08-21 at 10.13.53.png2976×2514 647 KB

Is there any kind of background task that would reduce the total number of segments increasing the total store size while leaving the respective primary values unaffected, as shown in the graphs?

Version: 6.8.0

2

the background task happening is a so-called merge. See https://www.elastic.co/guide/en/elasticsearch/reference/7.3/index-modules-merge.html

Just to rule this out, are there any open scroll contexts or snapshots going on that might leave files open?

3

Hello Alexander. Thank you for the quick reply. The response of GET _cluster/pending_tasks is "tasks" : [ ]. I have not used the scroll API on that index and have not provided an external API for anyone else to do so. The latest snapshot was taken on 2019-08-19.

4

Any chance you can share the output of

GET _stats?human
5

Of course. All: https://www.dropbox.com/s/ctn97bl95gyh97f/all.txt?dl=1 and the index in question: https://www.dropbox.com/s/6aqlc7nvs1ykqsg/index.txt?dl=1

6

can you also include some more parameters to get more stats for that index

GET your_index/_stats?human&level=shards&include_segment_file_sizes=true

Thanks!

7

Sure. https://www.dropbox.com/s/vlqe8o4feqv93mb/details.txt?dl=1 Thank you!

8

@kmousikos can you try running _flush&force=true&wait_if_ongoing=true followed by an _refresh and see if the issue persists

1 Like
9

Flushing the index solved the issue. Thank you very much for the assistance @s1monw .

1 Like
10

@kmousikos thanks for the feedback. I tried to address this to be less trappy here: https://github.com/elastic/elasticsearch/pull/46066

thanks for reporting this issue on the forum. I think we used to have protection for this in the past but it seemed to be a regression.

11

@s1monw my pleasure. Thanks for the support. Just for documentation completeness, I observed the same thing on that same index the following day, only that time the size of the primary was ~10GB (double the actual data) and the total size was ~15GB. After flushing again it returned to normal (5GB and 10GB) and has not displayed that behaviour since (it now is at 6.4GB, 12.7GB)

12

got it! I think it's the same issue. It's inconvenient for sure. Lets see if we can easily fix / address it to happen automatically.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.