While migrating index from ES2 to ES6 index size is gettting bigger

[manish_mitra]

indent preformatted text by 4 spaces
Hi,

We are upgrading Elastic stack from Elastic search 2.3.4 to elastic search 6.2.

When we are migrating index from ES2 to ES6, migrated index are taking more space in ES6 , we migrated thinking that ES6 would take less space. Need help if we are following right path.

Steps : We are taking snapshot from ES2. Then restoring it in ES5 and after reindex we are taking snapshot in ES5 and restoring them in ES6.

When we are reindexing in ES5 . Index of 16MB (in ES2)is taking 18 MB in ES5 . When we are restoring the reindexed index in ES6 it is taking 18 MB.

If any one can help why it is taking more space, or i am doing something wrong. When we are creating fresh index in ES6 they are taking less space.
indent preformatted text by 4 spaces

//**************************Mapping of index in ES2 Start ******************************//

{"unicalog-mt-2018.04.20":{"mappings":{"logs":{"_all":{"enabled":true,"omit_norms":true},"dynamic_templates":[{"string_fields":{"mapping":{"fielddata":

{"format":"disabled"},"index":"analyzed","omit_norms":true,"fields":{"raw":

{"index":"not_analyzed","type":"string"}},"type":"string"},"match":"*","match_mapping_type":"string"}}],"properties":{"@timestamp":

{"type":"date","format":"strict_date_optional_time||epoch_millis"},"@version":{"type":"string","index":"not_analyzed"},"CORRELATION_ID":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"comp_level":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"dc_name":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"file":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"geoip":{"dynamic":"true","properties":{"ip":

{"type":"ip"},"latitude":{"type":"float"},"location":{"type":"geo_point"},"longitude":{"type":"float"}}},"host":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"hostname":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"log_family":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"log_type":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"msg":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"path":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"platform":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"processname":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"text":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}},"timedelta":{"type":"string","norms":

{"enabled":false},"fielddata":{"format":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed"}}}}},"default":{"_all":

{"enabled":true,"omit_norms":true},"dynamic_templates":[{"string_fields":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"fields":

{"raw":{"index":"not_analyzed","type":"string"}},"type":"string"},"match":"*","match_mapping_type":"string"}}],"properties":{"@timestamp":

{"type":"date","format":"strict_date_optional_time||epoch_millis"},"@version":{"type":"string","index":"not_analyzed"},"geoip":{"dynamic":"true","properties":{"ip":

{"type":"ip"},"latitude":{"type":"float"},"location":{"type":"geo_point"},"longitude":{"type":"float"}}}}}}}}

//**************************Mapping of index in ES2 End ******************************//

//**************************Mapping of index in ES6 Start ******************************//

{"unicalog1-mt-2018.04.20":{"mappings":{"doc":{"properties":{"@timestamp":{"type":"date"},"@version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"CORRELATION_ID":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"comp_level":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"dc_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"file":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"hostname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"log_family":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"log_type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"msg":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"path":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"platform":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"processname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"text":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"timedelta":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}

//**************************Mapping of index in ES6 End ******************************//

[dadoonet (David Pilato)]

I moved your question to #elasticsearch.

This is not the right way to format properly your code:

Could you please use the preview window before posting?

As an example, you wrote:

{"unicalog-mt-2018.04.20":{"mappings":{"logs":{"_all":{"enabled":true,"omit_norms":true},"dynamic_templates":[{"string_fields":{"mapping":{"fielddata":

{"format":"disabled"},"index":"analyzed","omit_norms":true,"fields":{"raw":

It should be like:

{ 
  "unicalog-mt-2018.04.20" : {
    "mappings" : {
      "logs" : {
        "_all":{"enabled":true,"omit_norms":true},
        "dynamic_templates":[...]
...

To do that, just write:

```
{ 
  "unicalog-mt-2018.04.20" : {
    "mappings" : {
      "logs" : {
        "_all":{"enabled":true,"omit_norms":true},
        "dynamic_templates":[...]
...
```