# Reduce the amount of shards of index (Elasticsearch) **URL:** \ **Category:** Logstash\ **Created:** [April 23, 2019, 8:13pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109 "2019-04-23T20:13:21Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![zolthar-z](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zolthar-z/32/47895_2.png) [@zolthar-z](https://discuss.elastic.co/u/zolthar-z)\ **Post date:** [April 23, 2019, 8:13pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/1 "2019-04-23T20:13:22Z")
Hi All Right now I have a elk configuration where filebeat send logs to logstash and logstash applied a filter and send the information to elasticsearch, this configurations is generating to many shards on each index and some of this shards are unassigned. I was reading an a solution was reduce the amount of shard but I can't find how reduce the amount of shard for the current index and how apply this for the new indexes. the idea is decrease the amount of shards from 5 to 3. I was wondering if maybe one of you had the same issue and could solve it. Thanks in advance for your help. Regards. ---
**Author:** ![zolthar-z](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zolthar-z/32/47895_2.png) [@zolthar-z](https://discuss.elastic.co/u/zolthar-z)\ **Post date:** [May 2, 2019, 2:44pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/2 "2019-05-02T14:44:20Z")
Hi I was reading and one solution is reduce the amount of shard in logstash index, Do you know how to edit the template index for logstash to force the use of less amount of shards? Thanks. Regards. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 2, 2019, 2:57pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/3 "2019-05-02T14:57:43Z")
In elasticsearch you can edit a template using the [template API](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/indices-templates.html). So I'd read the existing template and then update it. Tomorrow it will be picked up when a new index will be created and your new settings will be applied. ---
**Author:** ![zolthar-z](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zolthar-z/32/47895_2.png) [@zolthar-z](https://discuss.elastic.co/u/zolthar-z)\ **Post date:** [May 22, 2019, 3:21pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/4 "2019-05-22T15:21:38Z")
Hi All before any change I update my elastic to 7.0.1 version, now the elasticsearch index for logstash has only 1 shard and have the same error "1 of 7 shards failed", checking the elasticsearch logs I found this error related with logstash index: "Fielddata is disabled on text fields by default. Set fielddata=true on [host.hostname] in order to load fielddata in memory by uninverting the inverted index." One solution could be modify the index but I create a new index every day so every day I have to modify the index, another solution could be modify the logstash template for elasticsearch but the true is that I don't know how to handle this, please your help if anyone know how manage this error. Thanks. Regards. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 22, 2019, 4:46pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/5 "2019-05-22T16:46:31Z")
I'm moving your question to #logstash as I believe you'll find a better help there. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 22, 2019, 6:09pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/6 "2019-05-22T18:09:43Z")
I had exact same error when I upgraded 1 of 5 shards failed. eventually it got clear by self I don't know what it was. basically on 7.x they have default to one shard per index. rather in in previous version it was 5 shard. reason seems valid. but I am not satisfy, though I am not an expert. post your logstash config file here and we should be able to find out how to reduce indice/shard count. ---
**Author:** ![zolthar-z](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zolthar-z/32/47895_2.png) [@zolthar-z](https://discuss.elastic.co/u/zolthar-z)\ **Post date:** [May 22, 2019, 7:26pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/7 "2019-05-22T19:26:11Z")
Hi Thanks for your answer, this is my logstash configuration: input { beats { port =\> 5044 } } filter { grok { match =\> ["message", "%{TIMESTAMP\_ISO8601:timestamp}"] } date { match =\> ["timestamp", "ISO8601"] } } output { elasticsearch { hosts =\> ["localhost:9200"] index =\> "%{[@metadata][beat]}-7-%{+YYYY.MM.dd}" } } ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 22, 2019, 7:37pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/8 "2019-05-22T19:37:11Z")
> [@zolthar-z](#): > > %{[@metadata][beat] Luise post some index name that appears in your systems. I am trying to see what %{[@metadata][beats]} is bringing the way I have configure is myindex-%{+YYYY.MM} and then index template = indexname-\* now I have myindex-2019.04, myindex-2019.05 and when I delete one index I have no problem on my vizulization ---
**Author:** ![zolthar-z](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zolthar-z/32/47895_2.png) [@zolthar-z](https://discuss.elastic.co/u/zolthar-z)\ **Post date:** [May 23, 2019, 7:16pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/9 "2019-05-23T19:16:53Z")
Hi Thanks for your response, I think I found the solution (a lucky break), I changed the way logstash send the information to elasticsearch adding the beat version: "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" With this change elasticasearch use filebeat \_index template instead logstash \_index template to save the data, so, elasticsearch save the data in a different way and the use of this index fix the problem. The root cause was the ignore\_above parameter of logstash \_index template, when checked the logstash index mapping found that this parameter for host.hostname was set on 254 **VS** filebeat index where this value was 1024. filebeat \_index GET filebeat-7.1.0-2019.05.23/\_mapping/field/host.hostname { "filebeat-7.1.0-2019.05.23" : { "mappings" : { "host.hostname" : { "full\_name" : "host.hostname", "mapping" : { "hostname" : { "type" : "keyword", "ignore\_above" : 1024 } } } } } } logstash \_index GET filebeat-7-2019.05.23/\_mapping/field/host.hostname { "filebeat-7-2019.05.23" : { "mappings" : { "host.hostname" : { "full\_name" : "host.hostname", "mapping" : { "hostname" : { "type" : "keyword", "ignore\_above" : 256 } } } } } } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 20, 2019, 7:16pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109/10 "2019-06-20T19:16:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.