# 300K file increases index diskspace with 400K
**URL:** https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310
**Category:** Elasticsearch
**Created:** [August 13, 2015, 8:43am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310 "2015-08-13T08:43:16Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@e\_ariens](https://discuss.elastic.co/u/e_ariens)
#### Post date: [August 13, 2015, 8:43am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/1 "2015-08-13T08:43:16Z")
Hi,
I have a question. We do a lot of work for acounting oficces. When testing we discoverd that a 300K file increased the index store with 400K. The file it self is not stored.
The file contain mainly numbers. I have some of the data attached to the question.
Please explain why a file with a lot of number increases the index that mutch.
Regards
Eric
> [@](#):
>
> Naam, Doc.ID, Gewijzigd op
> 4 - 00000001,2, 9-7-2015 12:00:00
> 5 - 00000002,3, 9-7-2015 12:00:00
> 6 - 00000003,4, 9-7-2015 12:00:00
> 7 - 00000004,5, 9-7-2015 12:00:01
> 8 - 00000005,6, 9-7-2015 12:00:01
> 9 - 00000006,7, 9-7-2015 12:00:01
> 10 - 00000007,8, 9-7-2015 12:00:01
> 11 - 00000008,9, 9-7-2015 12:00:01
> 12 - 00000009,10, 9-7-2015 12:00:01
> 13 - 00000010,11, 9-7-2015 12:00:01
> .
> .
> .
> .
> .
> .
> 396 - 00000019,394, 9-7-2015 12:00:41
> 397 - 00000020,395, 9-7-2015 12:00:41
> 398 - 00000021,396, 9-7-2015 12:00:41
> 399 - 00000022,397, 9-7-2015 12:00:41
> 400 - 00000023,398, 9-7-2015 12:00:41
> 401 - 00000024,399, 9-7-2015 12:00:41
> 402 - 00000025,400, 9-7-2015 12:00:42
> 404 - 00000001,402, 9-7-2015 12:00:43
> 405 - 00000002,403, 9-7-2015 12:00:43
> 406 - 00000003,404, 9-7-2015 12:00:43
> 407 - 00000004,405, 9-7-2015 12:00:43
> 408 - 00000005,406, 9-7-2015 12:00:43
> 409 - 00000006,407, 9-7-2015 12:00:43
> 410 - 00000007,408, 9-7-2015 12:00:43
> 411 - 00000008,409, 9-7-2015 12:00:43
> 412 - 00000009,410, 9-7-2015 12:00:43
> 413 - 00000010,411, 9-7-2015 12:00:43
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 13, 2015, 8:49am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/2 "2015-08-13T08:49:09Z")
What's the mapping of the type you're storing the data in?
It's common that data when indexed in ES requires more storage than the raw data.
---
### Author:  [@e\_ariens](https://discuss.elastic.co/u/e_ariens)
#### Post date: [August 13, 2015, 11:10am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/3 "2015-08-13T11:10:45Z")
{
"hyarchisdocument": {
"\_source": {
"excludes": [
"hyarchis\_attachment"
]
},
"\_id": {
"path": "hyarchis\_id"
},
"properties": {
"index1": {
"type": "integer",
"store": true
},
"index2": {
"type": "integer",
"store": true
},
"index3": {
"type": "integer",
"store": true
},
"index4": {
"type": "integer",
"store": true
},
"index5": {
"type": "string",
"store": true
},
"index6": {
"type": "string",
"store": true
},
"index7": {
"type": "integer",
"store": true
},
"index8": {
"type": "integer",
"store": true
},
"index9": {
"type": "integer",
"store": true
},
"index10": {
"type": "string",
"store": true
},
"index11": {
"type": "integer",
"store": true
},
"index12": {
"type": "integer",
"store": true
},
"index13": {
"type": "integer",
"store": true
},
"index14": {
"type": "string",
"store": true
},
"index15": {
"type": "string",
"store": true
},
"index16": {
"type": "integer",
"store": true
},
"index17": {
"type": "date",
"store": true
},
"index18": {
"type": "string",
"store": true
},
"index19": {
"type": "date",
"store": true
},
"index20": {
"type": "integer",
"store": true
},
"index1020": {
"type": "boolean",
"store": true
},
"index1021": {
"type": "integer",
"store": true
},
"index1022": {
"type": "string",
"store": true
},
"hyarchis\_documentid": {
"type": "integer",
"store": true
},
"hyarchis\_name": {
"type": "string",
"store": true
},
"hyarchis\_extensions": {
"type": "string"
},
"hyarchis\_attachment": {
"type": "attachment",
"store": false
}
}
}
}
---
### Author:  [@e\_ariens](https://discuss.elastic.co/u/e_ariens)
#### Post date: [August 13, 2015, 11:11am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/4 "2015-08-13T11:11:30Z")
This is the info stored:
{
"\_shard": 1,
"\_node": "3CyrcJqmQjm24IZoBtBpuw",
"\_index": "index1",
"\_type": "hyarchisdocument",
"\_id": "4680",
"\_score": 0.002076112,
"fields": {
"hyarchis\_name": [
"CijferSheet met comma als text"
],
"hyarchis\_documentid": [
4680
],
"hyarchis\_extensions": [
"txt"
],
"index11": [
78
],
"index1": [
1
],
"hyarchis\_id": [
4680
],
"index2": [
15
],
"index5": [
"1"
],
"index7": [
25
],
"index6": [
"Ariens"
],
"index18": [
"BAI\_1\_Static"
]
},
"sort": [
0.002076112
],
"\_explanation": {
"value": 0.002076112,
"description": "weight(\_all:4680 in 0) [PerFieldSimilarity], result of:",
"details": [
{
"value": 0.002076112,
"description": "fieldWeight in 0, product of:",
"details": [
{
"value": 1.7320508,
"description": "tf(freq=3.0), with freq of:",
"details": [
{
"value": 3,
"description": "termFreq=3.0"
}
]
},
{
"value": 0.30685282,
"description": "idf(docFreq=1, maxDocs=1)"
},
{
"value": 0.00390625,
"description": "fieldNorm(doc=0)"
}
]
}
]
}
}
---
### Author:  [@mosiddi](https://discuss.elastic.co/u/mosiddi)
#### Post date: [August 14, 2015, 11:59am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/5 "2015-08-14T11:59:02Z")
> [@e\_ariens](#):
>
> { "hyarchisdocument": { "\_source": { "excludes": ["hyarchis\_attachment"] },
You are storing \_source as well as indexing the fields and storing them. This will increase the size of index from what u have. If you don't want to support updates, u can just disable \_source and store fields u want to retrieve later. Your index size ideally should come down.
---
### Author:  [@e\_ariens](https://discuss.elastic.co/u/e_ariens)
#### Post date: [August 17, 2015, 5:33am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/6 "2015-08-17T05:33:47Z")
We do use updates so we can not disable \_source.
---
### Author:  [@mosiddi](https://discuss.elastic.co/u/mosiddi)
#### Post date: [August 17, 2015, 5:49am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/7 "2015-08-17T05:49:00Z")
I see. But now you know why your index diskspace is more than what you fed!
---
### Author:  [@e\_ariens](https://discuss.elastic.co/u/e_ariens)
#### Post date: [August 17, 2015, 6:30am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/8 "2015-08-17T06:30:54Z")
> [@mosiddi](#):
>
> But now you know why your index diskspace is more than what you fed!
Can you explain that further. The small amount of data we store in the \_source is not enough to explain the huge data. We do not store the file indexed
> [@e\_ariens](#):
>
> "\_source": { "excludes": ["hyarchis\_attachment"] }
---
### Author:  [@mosiddi](https://discuss.elastic.co/u/mosiddi)
#### Post date: [August 17, 2015, 8:14am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/9 "2015-08-17T08:14:33Z")
If you see the mapping, each field (except attachment) you are indexing, storing and also saving in \_source. While ES/Lucene creates inverted indices for the fields that are getting indexed, but storing/\_source will consume memory (though they will be compressed).
This is a good discussion on when to use store v/s \_source: [http://stackoverflow.com/questions/28678296/elasticsearch-store-field-vs-source](http://stackoverflow.com/questions/28678296/elasticsearch-store-field-vs-source) . One more good thread: [http://stackoverflow.com/questions/17103047/why-do-i-need-storeyes-in-elasticsearch](http://stackoverflow.com/questions/17103047/why-do-i-need-storeyes-in-elasticsearch)
Since u want partial update, \_source is needed. Question is why u want individual fields to be stored as well.
Regards,
Imran
Please note: I'm also learning ES and my participation in discussions help me learn more!
---
### Author:  [@e\_ariens](https://discuss.elastic.co/u/e_ariens)
#### Post date: [August 17, 2015, 8:38am UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/10 "2015-08-17T08:38:31Z")
I see your point. I could only use store. This will save me some diskspace.
But this stil leaves me with the huge disk usage for a file with a lot of numbers.
- A 300K file with mostly numbers will use 400K on disk.
- A 300K word document will only increase disk usage with 40K.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:55pm UTC](https://discuss.elastic.co/t/300k-file-increases-index-diskspace-with-400k/27310/11 "2017-07-05T23:55:35Z")