# Index size for files much bigger with ES5 compared to ES2

**URL:** <https://discuss.elastic.co/t/index-size-for-files-much-bigger-with-es5-compared-to-es2/73838>\
**Category:** Elasticsearch\
**Created:** [February 3, 2017, 11:51am UTC](https://discuss.elastic.co/t/index-size-for-files-much-bigger-with-es5-compared-to-es2/73838 "2017-02-03T11:51:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![David\_Pocivalnik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_pocivalnik/32/56098_2.png) [@David\_Pocivalnik](https://discuss.elastic.co/u/David_Pocivalnik)\
**Post date:** [February 3, 2017, 11:51am UTC](https://discuss.elastic.co/t/index-size-for-files-much-bigger-with-es5-compared-to-es2/73838/1 "2017-02-03T11:51:10Z")

</div>

I used mapper-attachments in order to index files. created the request to add it to field "file" and it's copied to the other fields. The following mapping definition was used with ES2

```
{
    "files": {
        "properties": {
            "startDate": {
                "type": "date", "index": "not_analyzed", "store": false, 
                "format": "yyyy-MM-dd HH:mm:ss:SSS||yyyy-MM-dd HH:mm:ss"

            },
            "mimetype": {
                "type": "integer", "index": "not_analyzed", "store": false
            },
            "file": { 
                "type": "attachment", 
                "fields": {
                    "title": { "store": false },
                    "content_type": { "store": false, "index": "no" },
                    "content": { "store": false, "term_vector": "with_positions_offsets", "type": "string", "copy_to": ["fileGrams", "fileEn", "fileLang1", "fileLang2"] },
                    "date": { "store": false },
                    "author": { "store": false },
                    "keywords": { "store": false },
                    "content_type" : { "store": false },
                    "language": { "store": false }
                }
            },
            "fileGrams": { 
                "type": "string", "index": "analyzed", "analyzer": "angram"
            },
            "fileEn": { 
                "type": "string", "index": "analyzed", "analyzer": "alangen"
            },
            "fileLang1": { 
                "type": "string", "index": "analyzed", "analyzer": "alang1"
            },
            "fileLang2": { 
                "type": "string", "index": "analyzed", "analyzer": "alang2"
            }
        }
    }
}

```

with ES5 I switched to ingest-attachment. Due to some changes with ES5 (no string field anymore and multi fields) the updated mapping looks like:

```
{
    "files": {
        "properties": {
            "startDate": {
                "type": "date", "index": true, "store": false, 
                "format": "yyyy-MM-dd HH:mm:ss:SSS||yyyy-MM-dd HH:mm:ss"
            },
            "mimetype": {
                "type": "integer", "index": true, "store": false
            },
            "file": { 
                "type": "text", "index": true,
                "fields": {
                    "fileGrams": { 
                        "type": "text", "index": true, "analyzer": "angram"
                    },
                    "fileEn": { 
                        "type": "text", "index": true, "analyzer": "alangen"
                    },
                    "fileLang1": { 
                        "type": "text", "index": true, "analyzer": "alang1"
                    },
                    "fileLang2": { 
                        "type": "text", "index": true, "analyzer": "alang2"
                    }
                }
            }
        }
    }
}

```

I now face the problem that with a defined set of data (around 120.000 documents, varying from email, pdf, xml etc.) the size of the index is much higher.  
ES2: 350MB  
ES5: 1800MB

when I remove the fields within "file" field (no multi fields):  
ES5: ~600MB

Any ideas/explanations what the reason(s) might be?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 3, 2017, 12:17pm UTC](https://discuss.elastic.co/t/index-size-for-files-much-bigger-with-es5-compared-to-es2/73838/2 "2017-02-03T12:17:27Z")

</div>

Hi David

That's super interesting. I'd not expect such difference.

Can you give the result of the exact mapping you have in 2.x and 5.x by running:

```auto
GET index/files/_mapping

```

Also can you give an example of a JSON document (`_source` field) in ES 2.x and in ES 5.x (please use the same document so I can really compare)?

And finally, can you run:

```auto
GET /yourindexname/_stats?include_segment_file_sizes&human

```

---

<div class="post-metadata">

**Author:** ![David\_Pocivalnik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_pocivalnik/32/56098_2.png) [@David\_Pocivalnik](https://discuss.elastic.co/u/David_Pocivalnik)\
**Post date:** [February 3, 2017, 1:19pm UTC](https://discuss.elastic.co/t/index-size-for-files-much-bigger-with-es5-compared-to-es2/73838/3 "2017-02-03T13:19:44Z")

</div>

see below for exact mappings.

providing the source fields takes until Monday (for ES2), I'll reply with both when I got them

the statistics are quite large, any particular part I shall provide?

ES2:

```
"mappings":{  
    "files":{  
        "properties":{  
            "startDate":{  
                "format":"yyyy-MM-dd HH:mm:ss:SSS||yyyy-MM-dd HH:mm:ss",
                "type":"date"
            },
            "mimetype":{  
                "type":"integer"
            },
            "fileGrams":{  
                "analyzer":"angram",
                "type":"string"
            },
            "fileEn":{  
                "analyzer":"alangen",
                "type":"string"
            },
            "file":{  
                "type":"attachment",
                "fields":{  
                    "date":{  
                        "type":"string"
                    },
                    "keywords":{  
                        "type":"string"
                    },
                    "content_type":{  
                        "type":"string"
                    },
                    "author":{  
                        "type":"string"
                    },
                    "name":{  
                        "type":"string"
                    },
                    "language":{  
                        "type":"string"
                    },
                    "title":{  
                        "type":"string"
                    },
                    "content":{  
                        "copy_to":[  
                            "fileGrams",
                            "fileEn",
                            "fileLang1",
                            "fileLang2"
                        ],
                        "term_vector":"with_positions_offsets",
                        "type":"string"
                    },
                    "content_length":{  
                        "type":"integer"
                    }
                }
            },
            "fileLang1":{  
                "analyzer":"alang1",
                "type":"string"
            },
            "fileLang2":{  
                "analyzer":"alang2",
                "type":"string"
            }
        }
    }
}

```

ES5

```
"mappings": {
  "files": {
    "properties": {
      "attachment": {
        "properties": {
          "content_length": {
            "type": "long"
          },
          "content_type": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "language": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "title": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "error": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "file": {
        "type": "text"
      },
      "mimetype": {
        "type": "integer"
      },
      "startDate": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss:SSS||yyyy-MM-dd HH:mm:ss"
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 3, 2017, 2:05pm UTC](https://discuss.elastic.co/t/index-size-for-files-much-bigger-with-es5-compared-to-es2/73838/4 "2017-02-03T14:05:16Z")

</div>

Can you change the properties for `content_type`, `language`, `title` fields to be `text` only and in ingest plugin remove `error` field?

I wonder what kind of value you can have in error field BTW.

---

<div class="post-metadata">

**Author:** ![David\_Pocivalnik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_pocivalnik/32/56098_2.png) [@David\_Pocivalnik](https://discuss.elastic.co/u/David_Pocivalnik)\
**Post date:** [February 6, 2017, 8:12am UTC](https://discuss.elastic.co/t/index-size-for-files-much-bigger-with-es5-compared-to-es2/73838/5 "2017-02-06T08:12:46Z")

</div>

the error field is because of my pipe definition

```
    cb.startObject()
            .startObject("attachment")
            .field("field", "data")
            .field("target_field", "attachment")
            .field("indexed_chars", "-1")
    // .startArray("on_failure")
    // .startObject()
    // .startObject("set")
    // .field("field", "error")
    // .field("value", "{{ _ingest.on_failure_message }} , {{ _ingest.on_failure_processor_type }}")
    // .endObject()
    // .endObject()
    // .endArray()
            .endObject()
            .endObject();
    cb.startObject()
            .startObject("set")
            .field("field", "file")
            .field("value", "{{ attachment.content }}")
            .field("ignore_failure", true)
            .endObject()
            .endObject();
    cb.startObject()
            .startObject("remove")
            .field("field", "attachment")
            .field("ignore_failure", true)
            .endObject()
            .endObject();
    cb.startObject()
            .startObject("remove")
            .field("field", "data")
            .field("ignore_failure", true)
            .endObject()
            .endObject();

```

as you can see I removed the error field and also everything within _attachment_ where the ingest plug-in writes the extracted data, the _data_ field gets removed as well (containing the base64 encoded data).

The size seems almost the same now with my data.

But I do not have any multi-fields for the content defining different analyzers at the moment.

So I'm wondering if I did it right with ES2 and the _copy\_to_ instruction, which should have copied the extracted content to the other fields I defined. For me it seems that my last config (with ES2) was wrong or did not work. I could understand that the index would need x times more space when having x times more fields with the same content but different analyzers.

Any ideas on that?

---

<div class="post-metadata">

**Author:** ![David\_Pocivalnik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_pocivalnik/32/56098_2.png) [@David\_Pocivalnik](https://discuss.elastic.co/u/David_Pocivalnik)\
**Post date:** [February 8, 2017, 9:13am UTC](https://discuss.elastic.co/t/index-size-for-files-much-bigger-with-es5-compared-to-es2/73838/6 "2017-02-08T09:13:15Z")

</div>

I just verified that the _copy-to_ did not work as I expected. Could you tell me what I was doing wrong?  
Here's the relevant mapping definition for ES 2.3.1

```
        "file": { 
            "type": "attachment", 
            "fields": {
                "title": { "store": false },
                "content_type": { "store": false, "index": "no" },
                "content": { "store": false, "term_vector": "with_positions_offsets", "type": "string", "copy_to": ["fileGrams", "fileEn", "fileLang1", "fileLang2"] },
                "date": { "store": false },
                "author": { "store": false },
                "keywords": { "store": false },
                "content_type" : { "store": false },
                "language": { "store": false }
            }
        },
        "fileGrams": { 
            "type": "string", "index": "analyzed", "analyzer": "angram"
        },
        "fileEn": { 
            "type": "string", "index": "analyzed", "analyzer": "alangen"
        },
        "fileLang1": { 
            "type": "string", "index": "analyzed", "analyzer": "alang1"
        },
        "fileLang2": { 
            "type": "string", "index": "analyzed", "analyzer": "alang2"
        }

```

I'm just curious why it didn't work and would love to know why.

Anyhow, my original question is obsolete, sorry for any misleading!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 8, 2017, 9:13am UTC](https://discuss.elastic.co/t/index-size-for-files-much-bigger-with-es5-compared-to-es2/73838/7 "2017-03-08T09:13:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.