# Elasticsearch Template Issue **URL:** \ **Category:** Elasticsearch\ **Created:** [November 26, 2018, 6:58am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121 "2018-11-26T06:58:45Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Madhukar](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@Madhukar](https://discuss.elastic.co/u/Madhukar)\ **Post date:** [November 26, 2018, 6:58am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/1 "2018-11-26T06:58:45Z")
I am creating the index template and define mapping in it. **Default Elasticsearch Settings** Sent data of size (on disk) 737 MB (173,170 Documents) to Elasticsearch, its size on elasticsearch is 320.1mb (default elasticsearch index setting) **Template Settings** Then I created a template and sent same data again. ```auto PUT /_template/example-name { "template" : "example-name*", "index_patterns" : [ "example-pattern*" ], "settings" : { "index" : { "number_of_shards" : "5", "number_of_replicas" : "0", "refresh_interval" : "5s" } } ``` Now the size is increased. Size = **956.2mb** Documents = 173,170 **Used Index Compression Settings** So i found an index setting for compression here [https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#\_static\_index\_settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#_static_index_settings) i.e. index.codec : best compression Used this settings: ```auto "settings" : { "index" : { "number_of_shards" : "5", "number_of_replicas" : "0", "refresh_interval" : "5s", "codec" : "best_compression" } ``` And now, size is 843.8mb. Could you please suggest how to do the index optimization and make sure the size reduces? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 26, 2018, 7:18am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/2 "2018-11-26T07:18:12Z")
Do you have the exact same scenario and data when you are doing your tests? To do a real comparison you should call at the end the `_forcemerge` API and merge to one single segment. My 2 cents ---
**Author:** ![Madhukar](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@Madhukar](https://discuss.elastic.co/u/Madhukar)\ **Post date:** [November 26, 2018, 7:29am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/3 "2018-11-26T07:29:25Z")
> [@dadoonet](#): > > `_forcemerge` Thanks for your response David. Yes, I have exact same scenario and data for this test. How do i use the `_forcemerge` API in the end while using template : ```auto PUT /_template/example-name/_forcemerge { "template" : "example-name*", "index_patterns" : [ "example-pattern*" ], "settings" : { "index" : { "number_of_shards" : "5", "number_of_replicas" : "0", "refresh_interval" : "5s", "codec" : "best_compression" } }, ``` I used this, but got illegal\_argument\_exception. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 26, 2018, 7:44am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/4 "2018-11-26T07:44:46Z")
Compression typically improves with shard size, so to make fair comparison I would recommend you to index into an index with a single primary shard. As the size on disk will fluctuate as segments are merged in the background, it is important to [force merge down to a single segment](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-forcemerge.html) once indexing has completed as the indices otherwise could be in varying stages of merging. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 26, 2018, 7:49am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/5 "2018-11-26T07:49:51Z")
I meant to call manually this API [https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) after you have injected all data. Then look at the size. As @Christian_Dahlqvist said, one single shard would be even better. ---
**Author:** ![Madhukar](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@Madhukar](https://discuss.elastic.co/u/Madhukar)\ **Post date:** [November 27, 2018, 11:08am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/6 "2018-11-27T11:08:31Z")
Thanks David, But that's not what we would want because if we use this manually, then we would need to do this for every index. The scenario here is that i want to create a template for a particular index pattern so that if any index that follows that particular pattern would be created automatically using the template settings. And I want dynamic mapping in it. ---
**Author:** ![Madhukar](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@Madhukar](https://discuss.elastic.co/u/Madhukar)\ **Post date:** [November 27, 2018, 11:13am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/7 "2018-11-27T11:13:46Z")
Thanks @Christian_Dahlqvist said, reducing to 1 shard helps to reduce size to some extent but not much useful. See the screenshot. It is still more than the original data ingested to Elasticsearch i.e. 173170 documents. ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/2/6289d17125d119b9b98cb662ceb1cdd8484fc1c4.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 27, 2018, 11:17am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/8 "2018-11-27T11:17:04Z")
I would recommend you have a look at the following resources and try optimise your mappings if you have not already done so: [https://www.elastic.co/guide/en/elasticsearch/reference/6.5/tune-for-disk-usage.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/tune-for-disk-usage.html) > **[Filebeat modules, access logs and Elasticsearch storage requirements |...](https://www.elastic.co/blog/filebeat-modiles-access-logs-and-elasticsearch-storage-requirements)** > > Elastic recently introduced Filebeat Modules, which are designed to make it extremely easy to ingest and gain insights from common log formats. These follow the principle that ---
**Author:** ![Madhukar](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@Madhukar](https://discuss.elastic.co/u/Madhukar)\ **Post date:** [November 27, 2018, 11:30am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/9 "2018-11-27T11:30:34Z")
Thanks @Christian_Dahlqvist for your response. We did the same thing today and it is working perfectly fine now. 🙂 Below is the screenshot with same number of documents and look at the size now. ![ELK_1](https://us1.discourse-cdn.com/elastic/original/3X/0/0/00a980ee17e69cbefc22345bb39c415b6158eda9.png) **Explanation:-** So I created a template with below settings: - Removed all my static fields that had type "keyword" from properties section of mapping. - Included only those static fields that i assigned a custom type like date, long, ip etc. - Any field except that would also be created as type "keyword". Because it is dynamic template. 1 shards 0 replicas codec =\> best compression - Below is the index setting: ![ELK_2](https://us1.discourse-cdn.com/elastic/original/3X/8/3/8353714e46660d5f87f92aebf9441a5118feb1e2.png) - Below is the dynamic template setting: ![ELK_3](https://us1.discourse-cdn.com/elastic/original/3X/d/e/de235508b387f320ea1da7172d8a8999c325d4e4.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 25, 2018, 11:30am UTC](https://discuss.elastic.co/t/elasticsearch-template-issue/158121/10 "2018-12-25T11:30:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.