Increased docs.count and store.size after Elasticsearch upgrade

Elastic Stack Elasticsearch
1

We have existing Elasticsearch 5.5 cluster and want to migrate to another one Elasticsearch 7+

Mapping is same for both Elasticsearch 5 and Elasticsearch 7 indices

    "mappings": {
        "properties": {
          "_all": {
            "enabled": false
          },
          "keywords_count": {
            "type": "integer"
          },
          "price_sum": {
            "type": "scaled_float",
            "scaling_factor": 100
          },
          "organic": {
            "type": "nested",
            "properties": {
              "position": {
                "type": "short"
              },
              "domain": {
                "type": "keyword"
              },
              "traffic": {
                "type": "integer"
              }
            }
          }
        }
      }

We fill the new index with remote reindex method. Both indices has same number of documents, returned by _cat/count method ( 20 mil ), but additional data from _cat/indices differs a lot

New version has bigger number of docs.count ( 2.6 bil on ES 7 vs 1.1bil on ES 5) and store.size ( 500 gb ES 7 vs 260 gb ES 5)

Cannot understand, what adds so much space requirement? Please help

2

Can you please provide the full output of the index stats API for both indices?

3

@Christian_Dahlqvist thanks for your reply!

Long output, please check the pastebin links

ES 5 _stats
https://pastebin.com/MLwKKdTU

ES 7 _stats
https://pastebin.com/6tx9MC0Y

4

I suppose there is an error in _cat/indices response from Elasticsearch 5.5 version

_cat/indices shows 268.7gb store.size
Index stats output shows 594.6gb store - and it's the same as real value on hdd

health status index      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   uk_2020_03 jVjreQwORMCQsmMUTpkb4g  40   0 1184215956      6269622    268.7gb        268.7gb

"store": {
        "size_in_bytes": 638463789329, //594.6gb
        "throttle_time_in_millis": 0
      }

ElasticSearch 7 stats and cat/indices output shows correct values

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.