Increase total fields limit via creation of index in logstash

[reswob]

My logstash conf has the following output:

output {
    elasticsearch {
         hosts => ["IP1:9200","IP2:9200"]
         index => "test-${+YYYY.MM.dd}"
         user => elastic
         password => supersecret
   }
}

The problem is that this particular log source causes the error saying the "limit of total fields [1000] in index" has been exceeded.

I found the following post:

Total Fields Limit setting

I have the 5.0.0-alpha3 setup here, sat alongside the live 2.3.3 version (on Ubuntu). There is a feeder Logstash which delivers the same data to both live and beta. On the beta, I have an error the the "limit of total fields [1000] in index" has been exceeded. Can anyone tell me how I set this limit? I've tried the setting "index.mapping.total_fields.limit: 2000" in the elasticsearch.yml, but this complains about an index setting in node settings. Thanks

which helped me fix it temporarily, but since a new index is created dynamically everyday, I have to run the command again every day... after I get to work and this means that a certain number of events prior to my arrival were NOT indexed due to this error.

Is there a way in the configuration of logstash to set the total fields limit when it creates the new index each day?

Thanks

EDIT: The correct answer has a follow up below, be sure to read the whole thread.

[theuntergeek (Aaron Mildenstein)]

It cannot be done directly in the Logstash configuration. You will have to put it in an index template. The default index template Logstash provides only matches indices with a logstash-* name pattern.

The current 5.x template can be found here.

As the setting is index.mapping.total_fields.limit: 2000, it should fit before or after line 5 in that example, leaving it looking something like:

  "settings" : {
    "index.refresh_interval" : "5s",
    "index.mapping.total_fields.limit": 2000
  },

You can take this file and edit it as needed, particularly line 2, which currently reads "template" : "logstash-*", should match your index name pattern.

As far as putting that new template in, you can do so with these Logstash template directives in the elasticsearch output plugin block:

  manage_template => true
  template => '/path/to/template.json'
  template_name => 'a_valid_name_here' # The default is logstash
  template_overwrite => true

[reswob]

OK, I created a template, let's call it new.json. I put it in a directory I created: /etc/logstash/templates
I changed the top of the example to which you linked to this:

{
  "template" : "new-*",
  "version" : 50001,
  "settings" : {
    "index.refresh_interval" : "5s"
    "index.mapping.total_fields.limit": 3000
},

Then in the config file I changed that to:

@reswob

output {
elasticsearch {
hosts => ["IP1:9200","IP2:9200"]
index => "test-${+YYYY.MM.dd}"
user => elastic
password => supersecret
manage_template => true
template => '/etc/logstash/templates/new.json'
template_name => 'new'
template_overwrite => true
}
}

And restarted logstash.

I got the following two lines:

[INFO] [logstash.output.elasticsearch] Using mapping template from {:path=> "/etc/logstash/templates/new.json"}
[ERROR] [logstash.output.elasticsearch] Failed to install template {:message=> unexpected character ('"'): was expecting comma to separate OBJECT entries\n at source [B@5c91381; line: 6, column: 6]".... (lots of debug trace log stuff)

Since I don't have a \ in any of the above config files or the json template file, I'm not sure what the error means. Or does it refer to something else going on?

(I'm concurrently working an issue with tech support regarding problems with x-pack monitoring logstash)

Thanks

[theuntergeek (Aaron Mildenstein)]

You omitted the , after the refresh_interval line, which I did have in my example above.

[reswob]

Will try that and let you know

[Christian_Dahlqvist]

@reswob

"template" : "new-*",

This template will apply to indices with names matching new-*. As you appear to be writing into an index that does not match this pattern, the template will not be applied.

[reswob]

@theuntergeek, that looks like it worked. I now have these three lines in the log (retyped here, but abbreviated)

Using mapping template from {:path=> “/etc/logstash/templates/new.json”}
Attempting to install template {:manage_template => {"template" => "new-*" .....
Installing elasticsearch template to _template/new .....

@Christian_Dahlqvist you are correct. Those names are the same in my actual configuration file. I mistyped them here.