# Increase total fields limit via creation of index in logstash **URL:** \ **Category:** Logstash\ **Created:** [August 29, 2017, 6:24pm UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742 "2017-08-29T18:24:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [August 29, 2017, 6:24pm UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742/1 "2017-08-29T18:24:20Z")
My logstash conf has the following output: ``` output { elasticsearch { hosts => ["IP1:9200","IP2:9200"] index => "test-${+YYYY.MM.dd}" user => elastic password => supersecret } } ``` The problem is that this particular log source causes the error saying the "limit of total fields [1000] in index" has been exceeded. I found the following post: > [@Total Fields Limit setting](https://discuss.elastic.co/t/total-fields-limit-setting/53004): > > I have the 5.0.0-alpha3 setup here, sat alongside the live 2.3.3 version (on Ubuntu). There is a feeder Logstash which delivers the same data to both live and beta. On the beta, I have an error the the "limit of total fields [1000] in index" has been exceeded. Can anyone tell me how I set this limit? I've tried the setting "index.mapping.total\_fields.limit: 2000" in the elasticsearch.yml, but this complains about an index setting in node settings. Thanks which helped me fix it temporarily, but since a new index is created dynamically everyday, I have to run the command again every day... after I get to work and this means that a certain number of events prior to my arrival were NOT indexed due to this error. Is there a way in the configuration of logstash to set the total fields limit when it creates the new index each day? Thanks EDIT: The correct answer has a follow up below, be sure to read the whole thread. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [August 29, 2017, 7:01pm UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742/2 "2017-08-29T19:01:07Z")
It cannot be done directly in the Logstash configuration. You will have to put it in an index template. The default index template Logstash provides only matches indices with a `logstash-*` name pattern. The _current_ 5.x template can be found [here](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/v8.1.1/lib/logstash/outputs/elasticsearch/elasticsearch-template-es5x.json). As the setting is `index.mapping.total_fields.limit: 2000`, it should fit before or after line 5 in that example, leaving it looking something like: ```auto "settings" : { "index.refresh_interval" : "5s", "index.mapping.total_fields.limit": 2000 }, ``` You can take this file and edit it as needed, particularly line 2, which currently reads `"template" : "logstash-*"`, should match _your_ index name pattern. As far as putting that new template in, you can do so with these Logstash template directives in the elasticsearch output plugin block: ```auto manage_template => true template => '/path/to/template.json' template_name => 'a_valid_name_here' # The default is logstash template_overwrite => true ``` ---
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [August 29, 2017, 7:52pm UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742/3 "2017-08-29T19:52:06Z")
OK, I created a template, let's call it new.json. I put it in a directory I created: /etc/logstash/templates I changed the top of the example to which you linked to this: ``` { "template" : "new-*", "version" : 50001, "settings" : { "index.refresh_interval" : "5s" "index.mapping.total_fields.limit": 3000 }, ``` Then in the config file I changed that to: > [@reswob](#): > > output { > elasticsearch { > hosts =\> ["IP1:9200","IP2:9200"] > index =\> "test-${+YYYY.MM.dd}" > user =\> elastic > password =\> supersecret > manage\_template =\> true > template =\> '/etc/logstash/templates/new.json' > template\_name =\> 'new' > template\_overwrite =\> true > } > } And restarted logstash. I got the following two lines: [INFO] [logstash.output.elasticsearch] Using mapping template from {:path=\> "/etc/logstash/templates/new.json"} [ERROR] [logstash.output.elasticsearch] Failed to install template {:message=\> unexpected character ('"'): was expecting comma to separate OBJECT entries\n at source [B@5c91381; line: 6, column: 6]".... (lots of debug trace log stuff) Since I don't have a \ in any of the above config files or the json template file, I'm not sure what the error means. Or does it refer to something else going on? (I'm concurrently working an issue with tech support regarding problems with x-pack monitoring logstash) Thanks ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [August 29, 2017, 7:53pm UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742/4 "2017-08-29T19:53:33Z")
You omitted the `,` after the `refresh_interval` line, which I did have in my example above. ---
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [August 29, 2017, 8:13pm UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742/5 "2017-08-29T20:13:32Z")
Will try that and let you know ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 30, 2017, 7:27am UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742/6 "2017-08-30T07:27:47Z")
> [@reswob](#): > > "template" : "new-\*", This template will apply to indices with names matching `new-*`. As you appear to be writing into an index that does not match this pattern, the template will not be applied. ---
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [August 30, 2017, 11:50am UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742/7 "2017-08-30T11:50:28Z")
@theuntergeek, that looks like it worked. I now have these three lines in the log (retyped here, but abbreviated) Using mapping template from {:path=\> “/etc/logstash/templates/new.json”} Attempting to install template {:manage\_template =\> {"template" =\> "new-\*" ..... Installing elasticsearch template to \_template/new ..... @Christian_Dahlqvist you are correct. Those names are the same in my actual configuration file. I mistyped them here. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 27, 2017, 11:50am UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742/8 "2017-09-27T11:50:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.