# Problem disk usage index (to large) **URL:** \ **Category:** Elasticsearch\ **Created:** [May 4, 2017, 9:00am UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513 "2017-05-04T09:00:53Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [May 4, 2017, 9:00am UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513/1 "2017-05-04T09:00:53Z")
Hello, I have problem with my elasticsearch when i upload data to ES. Intput log file : - 26 Go - 174 871 772 lines in log file Output (index in ES) : - 48 Go **(Increase of 12 Go)** - same lines (its ok) Here an exemple of log line : ``` 90|30768|3080368563|161E|1E4|059|04/2017 07:49|A0|LEGACYNT|WEBMAI|1010|3802|01|0000141|06|3538814700|00|WEBMAIL|WEBMAI|46524631 90|27638|2075468563|1E|2E|859|04/2017 17:49|A0|LEGACYNT|WEBMAI|1010|3802|01|0000141|06|3538174700|00|WEBMAIL|WEBMAI|46524631 90|30238|3035468563|1AE|2A14|085|04/2017 15:49|A0|LEGACYNT|WEBMAI|1010|3802|01|0000141|06|3564414700|00|WEBMAIL|WEBMAI|4654631 ``` As you can see, **somes fields are the same**. Also, i use csv filter to cut lines, and template to index my data, it looks like : ``` "templatelog1": { "order": 0, "version": 50001, "template": "log1-*", "settings": { "index": { "refresh_interval": "300s", "number_of_shards": "2", "translog": { "sync_interval": "5s", "durability": "async" }, "number_of_replicas": "0" } }, "mappings": { "_default_": { "dynamic_templates": [ { "string_fields": { "mapping": { "norms": false, "type": "text", "fields": { "keyword": { "index": "not_analyzed", "type": "keyword" } } }, "match_mapping_type": "string", "match": "*" } } ], ``` ... After i define mapping of numeric field (byte, short ...) What do you can talk me about this problem ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 4, 2017, 9:10am UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513/2 "2017-05-04T09:10:57Z")
You are storing every string field twice, as a text and as a keyword. Have you run a force merge on the index? > [@Beuhlet\_Reseau](#): > > "fields": { > "keyword": { > "index": "not\_analyzed", > "type": "keyword" > } You don't need the `not_analyzed`, if you set it as a keyword it does that automatically.[quote="Beuhlet\_Reseau, post:1, topic:84513"] "translog": { "sync\_interval": "5s", "durability": "async" } [/quote] Be very, very, very careful changing this. If you are happy with potential data loss then you don't need to change anything. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 4, 2017, 9:29am UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513/3 "2017-05-04T09:29:49Z")
[This blog post](https://www.elastic.co/blog/elasticsearch-storage-the-true-story-2.0) may be a bit old, but is still quite useful. Enabling `best_compression` is an easy way to save some space, but does result in slightly higher CPU usage. Not having a lot of very small shards is also likely to help improve compression ratios. ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [May 9, 2017, 12:58pm UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513/4 "2017-05-09T12:58:35Z")
> [@warkolm](#): > > You are storing every string field twice, as a text and as a keyword It's possible to disable it @warkolm We use only keyword with Visualize. > [@warkolm](#): > > Be very, very, very careful changing this Yes, but There was no commit at each write, it's every 5 secondes. Indexing is faster. About your link @Christian_Dahlqvist : > [@Christian\_Dahlqvist](#): > > This blog post may be a bit old, but is still quite useful The use case n°3 : String : not\_analyzed and \_all : disabled. =\> How not analyzed all string ? To disable \_all it's just : ``` "_all": { "norms": false, "enabled": false} ``` Thank you ! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2017, 1:36pm UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513/5 "2017-05-09T13:36:40Z")
In Elasticsearch 5.x `not_analyzed` corresponds to `keyword` mapping fro string fields. ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [May 10, 2017, 9:23am UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513/6 "2017-05-10T09:23:32Z")
@Christian_Dahlqvist You means that not analyzed is by default ? I don't understand. =\> The twice storage of fields (string + keyword) is mandatory ? =\> String is always not analyzed if i have keyword =\> If i don't want analyze few keywords, it's possible ? I have disable \_all and **i have a rate of decrease about 30% of space disk**. =\> \_All field is useless i hope 😕 (I do not understand very well explain in documentation...) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 10, 2017, 1:34pm UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513/7 "2017-05-10T13:34:18Z")
> [@Beuhlet\_Reseau](#): > > =\> The twice storage of fields (string + keyword) is mandatory ? This is controlled through mappings and can be defined based on how you need to be able to query the data. > [@Beuhlet\_Reseau](#): > > =\> String is always not analyzed if i have keyword Yes, the string field mappings were updated in Elasticsearch 5.x and the old `not_analyzed` setting was replaced by the `keyword` mapping. > [@Beuhlet\_Reseau](#): > > =\> If i don't want analyze few keywords, it's possible ? I am not sure I understand what you mean. Through the mappings you can control how you want to index and analyze the fields and you can even configure fields to not be indexed at all. > [@Beuhlet\_Reseau](#): > > \_All field is useless i hope The \_all field is not useless. If you use Kibana and add filtering through the search bar without specifying. specific field, it is the \_all field that is used behind the scenes. For some use cases this is a perfectly acceptable trade-off, but it is not the right thing for everyone, so check if/how it affects the user experience. ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [May 10, 2017, 2:44pm UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513/8 "2017-05-10T14:44:48Z")
OK thank you @Christian_Dahlqvist with link i have win about 46% space disk. Do you know if it's possible to : =\> Remove index which have 3 days old for example =\> Or move index in zip folder to reuse them later in other ELK Stack. The solution is very demanding in disk space so, i want clear index automatically if it's have 3 days old. (Just a test) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 7, 2017, 2:53pm UTC](https://discuss.elastic.co/t/problem-disk-usage-index-to-large/84513/9 "2017-06-07T14:53:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.