# Limit of total fields \[1000\] in index \[p\_bs\_api\_abcd\_2020\_23\] has been exceeded **URL:** \ **Category:** Logstash\ **Created:** [June 3, 2020, 6:41am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476 "2020-06-03T06:41:41Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Saravana\_Maadavan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravana_maadavan/32/60232_2.png) [@Saravana\_Maadavan](https://discuss.elastic.co/u/Saravana_Maadavan)\ **Post date:** [June 3, 2020, 6:41am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/1 "2020-06-03T06:41:42Z")
I have followed the steps discussed in link [Increase total fields limit via creation of index in logstash](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742) but not able to resolve it. Kindly find below few snippets of the configuration files. > output { > elasticsearch { > hosts =\> ["XX.XX.XX.XX:7001"] > index =\> "%{Service}\_%{+YYYY\_ww}" > manage\_template =\> true > template =\> '/config/etc/logstash/templates/api\_template.json' > template\_name =\> 'api' > template\_overwrite =\> true > } > } Template json as follows: > { > "index\_patterns" : "p\_bs\_api\_abcd\*", > "version" : 60001, > "settings" : { > "index.refresh\_interval" : "5s", > "number\_of\_shards": 1, > "index.mapping.total\_fields.limit": 2000 > }, > "mappings" : { > "dynamic\_templates" : [ { > "message\_field" : { > "path\_match" : "message", > "match\_mapping\_type" : "string", > "mapping" : { > "type" : "text", > "norms" : false > } > } > }, { > "string\_fields" : { > "match" : "\*", > "match\_mapping\_type" : "string", > "mapping" : { > "type" : "text", "norms" : false, > "fields" : { > "keyword" : { "type": "keyword", "ignore\_above": 256 } > } > } > } > } ], > "properties" : { > "@timestamp": { "type": "date"}, > "@version": { "type": "keyword"}, > "geoip" : { > "dynamic": true, > "properties" : { > "ip": { "type": "ip" }, > "location" : { "type" : "geo\_point" }, > "latitude" : { "type" : "half\_float" }, > "longitude" : { "type" : "half\_float" } > } > } > } > } > } When I restart the logstash still I am able to see error as below. > [2020-06-03T07:08:17,196][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>"/mnt/storage/software/logstash-7.0.0/config/etc/logstash/templates/api\_template.json"} > [2020-06-03T07:08:17,293][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{" **index\_patterns"=\>"logstash-\*"** , "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s", "number\_of\_shards"=\>1, "index.mapping.total\_fields.limit"=\>2000, "index.lifecycle.name"=\>"logstash-policy", "index.lifecycle.rollover\_alias"=\>"logstash"}, "mappings"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"\*", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}} > [2020-06-03T07:08:17,321][INFO][logstash.outputs.elasticsearch] Installing elasticsearch template to \_template/api > [2020-06-03T07:12:16,988][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"p\_bs\_api\_abcd\_2020\_23", :\_type=\>"\_doc", :routing=\>nil}, #LogStash::Event:0x4b6e8b56], :response=\>{"index"=\>{"\_index"=\>"p\_bs\_api\_abcd\_2020\_23", "\_type"=\>"\_doc", "\_id"=\>"GXjPeHIBIVr3QtguFT6g", "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"**Limit of total fields [1000] in index [p\_bs\_api\_abcd\_2020\_23] has been exceeded"}}}}** @theuntergeek @Christian_Dahlqvist - Have looped you in since you guys have solved the issue in the thread I followed so kindly pointing out where I have went wrong. Cheers, Maadavan ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 3, 2020, 7:10am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/2 "2020-06-03T07:10:48Z")
The index\_pattern is wrong and what you provided in the template does not match your index name so does not get applied. ---
**Author:** ![Saravana\_Maadavan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravana_maadavan/32/60232_2.png) [@Saravana\_Maadavan](https://discuss.elastic.co/u/Saravana_Maadavan)\ **Post date:** [June 3, 2020, 9:11am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/3 "2020-06-03T09:11:21Z")
@Christian_Dahlqvist, Thanks for reverting, but index pattern do match isn't it? In the template I have given it as > p\_bs\_api\_abcd\* whereas in the error index name is **p\_bs\_api\_abcd\_2020\_23** in the error logs. Year & week number at the end of index name is dynamic and would be taken care of wildcard symbol (\*) at the end of index name given in template. Cheers, Maadavan ---
**Author:** ![richylyq](https://avatars.discourse-cdn.com/v4/letter/r/41988e/32.png) [@richylyq](https://discuss.elastic.co/u/richylyq)\ **Post date:** [June 3, 2020, 9:13am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/4 "2020-06-03T09:13:43Z")
> [@Saravana\_Maadavan](#): > > "index.mapping.total\_fields.limit": 2000 hi, try increasing the digits here? not sure if that is the issue here ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 3, 2020, 9:37am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/5 "2020-06-03T09:37:19Z")
What is the output of \_cat/templates ? ---
**Author:** ![Saravana\_Maadavan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravana_maadavan/32/60232_2.png) [@Saravana\_Maadavan](https://discuss.elastic.co/u/Saravana_Maadavan)\ **Post date:** [June 3, 2020, 11:01am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/6 "2020-06-03T11:01:26Z")
@Christian_Dahlqvist, Kindly find the snippet of the template in ES. Still could see index patterns are as logstash-\* instead of "api" which I have given it in template (snippet is present in my question post). Also I could see field limit has increased. So as you said it may be due to index pattern not matching. Where would I be giving that pattern? > { > "api" : { > "order" : 0, > "version" : 60001, > "index\_patterns" : [ > "logstash-\*" > ], > "settings" : { > "index" : { > "lifecycle" : { > "name" : "logstash-policy", > "rollover\_alias" : "logstash" > }, > "mapping" : { > "total\_fields" : { > "limit" : "2000" > } > }, > "refresh\_interval" : "5s", > "number\_of\_shards" : "1" > } > } @richylyq Thanks for your reply. I am able to see the updated field in template as I said above and the error is still only for 1000. Cheers, Maadavan ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 3, 2020, 11:04am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/7 "2020-06-03T11:04:09Z")
Look at the index\_patterns field which does not match the name of the index you are using. ---
**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Post date:** [June 3, 2020, 11:13am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/8 "2020-06-03T11:13:37Z")
> [@Saravana\_Maadavan](#): > > "index\_patterns" : "p\_bs\_api\_abcd\*" As per the documentation, the `index_patterns` in the index template needs to be an array of strings, not just a string as you have applied. So try setting that to `"index_patterns":["p_bs_api_abcd*"]` See [here.](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html#put-index-template-api-request-body) ---
**Author:** ![Saravana\_Maadavan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravana_maadavan/32/60232_2.png) [@Saravana\_Maadavan](https://discuss.elastic.co/u/Saravana_Maadavan)\ **Post date:** [June 5, 2020, 8:25am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/9 "2020-06-05T08:25:05Z")
Thanks Rahul. I made the changes in template to array format , even then the \_cat/templates shows index pattern to the default value logstash-\*. As per [Logstash 7.0 does ignore index\_patterns / template fields in template and overrules that with "logstash-\*"](https://discuss.elastic.co/t/logstash-7-0-does-ignore-index-patterns-template-fields-in-template-and-overrules-that-with-logstash/179341), seems there to be a bug in logstash 7.0. So have added **ilm\_enabled =\> false** in my logstash output. > output { > elasticsearch { > hosts =\> ["localhost:61000"] > index =\> "%{capability}\_%{+YYYY\_ww}" > ilm\_enabled =\> false > manage\_template =\> true > template =\> '/config/etc/logstash/templates/increased\_field\_limit.json' > template\_name =\> 'increased\_field\_limit' > template\_overwrite =\> true > } > } Post the above changes, in \_cat/templates I am able to see the index pattern has been updated and matching with the index names too. But still I am receiving the error "reason"=\>"Limit of total fields [1000] in index [XXXXX] has been exceeded" Is it being caused by ilm\_enabled =\> false or anything else to be checked? Or a version upgrade would solve the issue? Or should I reindex? Any help much appreciated please. Cheers, Maadavan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 3, 2020, 8:25am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476/10 "2020-07-03T08:25:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.