Is it expected to have different versions in the `*_LuceneXX_X.*` filenames?

Elastic Stack Elasticsearch
1

Hi

This is mostly out of curiosity and not as a question tied to a specific problem.

I was looking at the actual directories and files of some ES indices, I noticed a bunch of lucene files, but what got my attention is some files such as _2c_Lucene80_0.dvd and _2c_Lucene84_0.doc. The 80_0 and 84_0 seem like Lucene versions but I could be wrong. We are using ES version 7.10.1, and have upgraded to this version from older 6.x and 7.x versions. So I guess it's expected that the Lucene data went through different versions of Lucene.

If these numbers are indeed Lucene versions, I'd like to ask if this is:
a) expected?
b) does this imply there is some kind of "optimization" I can do for these indices?

/mnt/elasticsearch_0/nodes/0/indices/zmpTxCA2Tb2nLkK3RwZHdA$ ls -alh 0/index/*Lucene*
-rw-r--r-- 1 elasticsearch elasticsearch 591M Feb 21  2022 0/index/_2c_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch 3.9K Feb 21  2022 0/index/_2c_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 256M Feb 21  2022 0/index/_2c_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch 105M Feb 21  2022 0/index/_2c_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 542M Feb 21  2022 0/index/_2c_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch 9.7M Feb 21  2022 0/index/_2c_Lucene84_0.tip
-rw-r--r-- 1 elasticsearch elasticsearch 2.4K Feb 21  2022 0/index/_2c_Lucene84_0.tmd
-rw-r--r-- 1 elasticsearch elasticsearch 1.1G Feb 21  2022 0/index/_36_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch 3.7K Feb 21  2022 0/index/_36_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 518M Feb 21  2022 0/index/_36_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch 183M Feb 21  2022 0/index/_36_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 915M Feb 21  2022 0/index/_36_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch  21M Feb 21  2022 0/index/_36_Lucene84_0.tip
-rw-r--r-- 1 elasticsearch elasticsearch 2.5K Feb 21  2022 0/index/_36_Lucene84_0.tmd
-rw-r--r-- 1 elasticsearch elasticsearch 525M Feb 21  2022 0/index/_48_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch 3.5K Feb 21  2022 0/index/_48_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 226M Feb 21  2022 0/index/_48_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch  94M Feb 21  2022 0/index/_48_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 482M Feb 21  2022 0/index/_48_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch 9.5M Feb 21  2022 0/index/_48_Lucene84_0.tip
-rw-r--r-- 1 elasticsearch elasticsearch 2.6K Feb 21  2022 0/index/_48_Lucene84_0.tmd
2

You could try a force merge on all your indices, that would make Lucene recreate things in the latest version. Otherwise, I wouldn't be super worried.

3

Thanks for the suggestion.

It seems that force merge doesn't change the situation. Again, I think I see 2 lucene versions due to the Lucene80_0 and Lucene84_0 parts in the filenames, I am not sure if this is indeed the case.

The index I was looking at was closed.

$ curl -s 'localhost:9200/_cat/indices?v' | grep Y9VTS1sgThK6Jicvt8_Z2A
green  close  foo.boo-2020          Y9VTS1sgThK6Jicvt8_Z2A   1   1 

$ curl -s 'localhost:9200/_cat/shards?' | grep foo.boo-2020
foo.boo-2020          0 p STARTED  11171420    1.7gb 10.40.22.115 ip-10-40-22-115
foo.boo-2020          0 r STARTED  11171420    1.7gb 10.40.21.165 ip-10-40-21-165

The initial state with I think is 2 lucene versions

ip-10-40-21-165 $ ls -al /mnt/elasticsearch_*/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/*Lucene*
-rw-r--r-- 1 elasticsearch elasticsearch 43429780 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch     2151 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 24150842 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch   658361 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 38382222 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch   700546 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.tip
-rw-r--r-- 1 elasticsearch elasticsearch 40424293 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch     2151 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 22987332 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch   622078 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 36119317 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch   584489 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.tip

ip-10-40-22-115 $ ls -al /mnt/elasticsearch_*/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/*Lucene*
-rw-r--r-- 1 elasticsearch elasticsearch 43429780 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch     2151 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 24150842 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch   658361 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 38382222 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch   700546 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.tip
-rw-r--r-- 1 elasticsearch elasticsearch 40424293 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch     2151 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 22987332 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch   622078 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 36119317 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch   584489 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.tip

I open the index

$ curl -X POST "localhost:9200/foo.boo-2020/_open?pretty"
{
  "acknowledged" : true,
  "shards_acknowledged" : true
}

Then force merge it

$ curl -X POST "localhost:9200/foo.boo-2020/_forcemerge?pretty"
{
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  }
}

just dumping the setting in case it helps

$ curl "localhost:9200/foo.boo-2020/_settings?pretty"
{
  "foo.boo-2020" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "keep_forever_move_old"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_warm,data_hot"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "foo.boo-2020",
        "creation_date" : "1594598415497",
        "unassigned" : {
          "node_left" : {
            "delayed_timeout" : "15m"
          }
        },
        "number_of_replicas" : "1",
        "uuid" : "Y9VTS1sgThK6Jicvt8_Z2A",
        "version" : {
          "created" : "7070199",
          "upgraded" : "7100199"
        }
      }
    }
  }
}

no change at all

ip-10-40-22-115 $ ls -al /mnt/elasticsearch_*/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/*Lucene*
-rw-r--r-- 1 elasticsearch elasticsearch 38382222 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch   700546 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.tip
-rw-r--r-- 1 elasticsearch elasticsearch 40424293 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch     2151 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 22987332 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch   622078 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 36119317 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch   584489 Nov 29  2021 /mnt/elasticsearch_0/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.tip

ip-10-40-21-165 $  ls -al /mnt/elasticsearch_*/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/*Lucene*
-rw-r--r-- 1 elasticsearch elasticsearch 43429780 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch     2151 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 24150842 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch   658361 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 38382222 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch   700546 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_2l_Lucene84_0.tip
-rw-r--r-- 1 elasticsearch elasticsearch 40424293 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene80_0.dvd
-rw-r--r-- 1 elasticsearch elasticsearch     2151 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene80_0.dvm
-rw-r--r-- 1 elasticsearch elasticsearch 22987332 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.doc
-rw-r--r-- 1 elasticsearch elasticsearch   622078 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.pos
-rw-r--r-- 1 elasticsearch elasticsearch 36119317 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.tim
-rw-r--r-- 1 elasticsearch elasticsearch   584489 May 12 05:37 /mnt/elasticsearch_1/nodes/0/indices/Y9VTS1sgThK6Jicvt8_Z2A/0/index/_41_Lucene84_0.tip
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.