Index.mapping.total_fields.limit

[Bruceclegg]

...
We're running elasticsearch 6.8.15

For various reasons we are unable to run Lifecycle Management on our indexes - so I have set up a cron job which uses curator that when an index reaches a certain size or age it will be deleted (and recreated).

I am having a problem figuring out how to automate the index.mapping.total_fields.limit. A couple of our indexes require a limit > 1000. We've been running them like that for ages and we haven't experienced any performance issues. Currently, after curator runs I go to the gui and add the limit manually by editing the settings from the management page. But this only updates the settings for the current index and when the image is recreated the next time I have to do it again.

What is the best way to override the default index.mapping.total_fields.limit at index creation?
...

[theuntergeek (Aaron Mildenstein)]

Using that setting in an index template will do the trick.

[Bruceclegg]

Thank you for your reply. I have been out for two weeks with a nasty case of Covid. It sucked.

I did not set up the templates here - but what I propose to do is from the console run:

PUT _template/aacp-log-test
{
"mapping": {
"total_fields": {
"limit": "2500"
}
}

Am I on the right track?

[Bruceclegg]

You are being very helpful. I do appreciate it.

PUT _template/aacp-log-test
{
"mapping": {
"total_fields": {
"limit": "2500"
}
}}

yields:

{
"error": {
"root_cause": [
{
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: index patterns are missing;"
}
],
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: index patterns are missing;"
},
"status": 400
}

if I change "mapping" to "mappings" I get:

{
"error": {
"root_cause": [
{
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: index patterns are missing;"
}
],
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: index patterns are missing;"
},
"status": 400
}

I've tried some other variations, but I'm still getting the Validation Failed message. Can you help me understand the error message better please?

[Bruceclegg]

I tried this:

PUT _template/aapc.log-test
{
"index_patterns" : ["aapc.log-tes*"],
"mappings": {
"total_fields": {
"limit": "2500"
}
}}

and got:

{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [limit : 2500]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [total_fields]: Root mapping definition has unsupported parameters: [limit : 2500]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [limit : 2500]"
}
},
"status": 400
}

[leandrojmp (Leandro Pereira)]

The index.mapping.total_fields.limit is a setting, it needs to be inside the settings block of the index template, you are putting it inside the mappings block.

Try this:

PUT _template/aapc.log-test
{
   "index_patterns" : [ "aapc.log-tes*" ],
   "settings" : {
     "index" : {
       "mapping" : {
         "total_fields" : {
           "limit": "2500"
         }
       }
     }
   }
}

[Bruceclegg]

@leandrojmp

PUT _template/aapc.log-test
{
   "index_patterns" : [ "aapc.log-tes*" ],
   "settings" : {
     "index" : {
       "mapping" : {
         "total_fields" : {
           "limit": "2500"
         }
       }
     }
   }
}

Thank you - this worked perfectly.