# Storage gains in removing redundant message fields **URL:** \ **Category:** Elasticsearch\ **Created:** [December 6, 2018, 10:04pm UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812 "2018-12-06T22:04:14Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 6, 2018, 10:04pm UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/1 "2018-12-06T22:04:14Z")
Consider the following document (no logstash is involved here): ``` { "_index": "day-wise-2018.11.25", "_type": "type1", "_id": "11111111111111111111111111111111111111111111111111111111", "_score": 1, "_source": { "id": "012345678910111213", "LOAD_AVG_MIN": 1.000001, "ts": "2018-11-25T01:41:04.045Z", "u": "Load", "foo_field": "abcdef-ghi-jk-lmnopq-rstu-v-release_181010_270", "@id": "11111111111111111111111111111111111111111111111111111111", "@timestamp": "2018-11-25T01:41:04.221Z", "@message": "{\"id\":\"012345678910111213\",\"LOAD_AVG_MIN\":1.000001,\"ts\":\"2018-11- 25T01:41:04.045Z\",\"u\":\"Load\",\"foo_field\":\"abcdef-ghi-jk-lmnopq-rstu-v-release_181010_270\"}", "@owner": "foo_owner", "@log_group": "type1", } } ``` As you can see, the `@message` field looks to be redundant here since the contents of it are indexed as individual K,V pairs. for e.g the `id`, `LOAD_AVG_MIN`, `ts` etc are all indexed are separate fields (and their mappings too are defined). 1. Will it result in considerable reduction in size if I do away with the `@message` field? In the example above the contents of the field are pretty small but it can be large as well 2. I'm also thinking to do away with the redundant `@id` and `@log_group` fields which have the same values as `_id` and `_type` respectively. Does it yield any benefit? ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 10, 2018, 7:17pm UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/2 "2018-12-10T19:17:49Z")
@warkolm - any suggestions here? ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 19, 2018, 3:28am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/3 "2018-12-19T03:28:36Z")
@Christian_Dahlqvist any thoughts here? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 19, 2018, 7:28am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/4 "2018-12-19T07:28:54Z")
Yes, that can reduce how much space your data takes up on disk. It is hard to tell how much difference it will make so I would recommend that you test it. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 19, 2018, 7:40am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/5 "2018-12-19T07:40:56Z")
Thanks @Christian_Dahlqvist. One more question on storage gains is: does it save considerable storage if we set `enabled: false` i.e store the field(s) but not index it (and thereby not make them searchable). ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 19, 2018, 7:42am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/6 "2018-12-19T07:42:03Z")
Yes, that is another way to save space in case you need to keep the raw data. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 19, 2018, 7:42am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/7 "2018-12-19T07:42:58Z")
Thank you. Will do a quick POC with `enabled: false` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 19, 2018, 7:47am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/8 "2018-12-19T07:47:16Z")
As disk usage can vary depending on how much data you have per shard and where in the merging cycle you are, I would recommend indexing a reasonably large amount of data into an index with a single shard and then force merge this down to a single segment at the end. That will ensure you are doing as fair a comparison as possible. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 19, 2018, 7:50am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/9 "2018-12-19T07:50:04Z")
You mean to say that we first `forcemerge` and see the disk occupied and then index with `enabled:false`, run `forcemerge`and then again check the disk occupied to get a fair comparison? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 19, 2018, 7:59am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/10 "2018-12-19T07:59:36Z")
Index the data into an index with a single primary shard with the current mappings. Then index the same data into a separate index with a single shard using the updated mappings. Then forcemerge both indices down to a single segment and compare the size. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 19, 2018, 8:01am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/11 "2018-12-19T08:01:17Z")
Thank you for the excellent clarification. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 16, 2019, 8:01am UTC](https://discuss.elastic.co/t/storage-gains-in-removing-redundant-message-fields/159812/12 "2019-01-16T08:01:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.