Hello,
We are attempting to set the index.mapping.depth.limit to 3 for a new index template as we are seeing dynamically created objects that keep adding new fields (4,000+). Is there a way to set the depth limit in the composable index template, or do we have to do it through the API?
We've tried both methods and cannot get them to work, possibly a syntax issue with the API and/or the composable index settings. This is a short-term solution for a data ingestion we are experiencing, but trying to handle it on our end in the short-term.
Does anyone have any examples? Here's what we've tried:
PUT _template/my_index_template
{
"index_patterns": [ "index_pattern*" ],
"mappings": {
"index.mapping.depth.limit": "3"
}
}
This gives us an error "Root mapping definition has unsupported parameters: [index.mapping.depth.limit : 3]".
Similarly in the composable index, the preview shows a 400 error, so this isn't working there either.
Thanks in advance!
Hey @brandon.n,
Your example is close! The depth limit is a setting not a mapping. So for your example try something like this:
PUT _template/my_index_template
{
"index_patterns": [ "index_pattern*" ],
"settings": {
"index.mapping.depth.limit": "3"
}
}
Thanks @Kathleen_DeRusso!
That worked. Additionally, it looks like the only way to tell aside from the data coming in is using GET _template/my_index_template correct? It seems there's a difference between "_index_template" and "_template" for the API, but I don't see a ton of documentation or examples.
Thanks again!
You want to double check that the setting is in the template and the template is being applied? Correct, you can perform a GET request on the template to view the template or a GET index_name to view the settings and other index configuration after the template has already been applied at index creation time.
As far as _template vs. _index_template, _template is a legacy index template API, newer versions of Elasticsearch would be better suited to use composable templates and the _index_template APIs.
Okay thanks again. I asked primarily the difference from _template vs _index_template because running GET on those for the same template reveals different info. Currently we're on v8.10.2 and if I do GET _template/my_template, it reveals more relevant template information for what I was asking above, specifically it confirms the index.mapping.depth.limit and index.mapping.ignore_malformed settings, whereas if I do GET _index_template/my_template, it only gives the settings from the composable index settings, but I can't confirm if the above settings are acknowledged: TRUE.
You should be able to see this information.
Here's a simple example - I did this on an 8.10.2 cluster to be sure you can duplicate it:
PUT _index_template/template_1
{
"index_patterns": ["new-index-template*"],
"template": {
"settings": {
"index.mapping.depth.limit": "3"
}
}
}
Now, when you try to view the template:
GET _index_template/template_1
it returns
{
"index_templates": [
{
"name": "template_1",
"index_template": {
"index_patterns": [
"new-index-template*"
],
"template": {
"settings": {
"index": {
"mapping": {
"depth": {
"limit": "3"
}
}
}
}
},
"composed_of": []
}
}
]
}
Next I created a new index: PUT new-index-template-test
And GET new-index-template-test returns:
{
"new-index-template-test": {
"aliases": {},
"mappings": {},
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"mapping": {
"depth": {
"limit": "3"
}
},
"number_of_shards": "1",
"provided_name": "new-index-template-test",
"creation_date": "1729624137249",
"number_of_replicas": "1",
"uuid": "vFPfq6VMR_-v4kJbtQFzzQ",
"version": {
"created": "8100299"
}
}
}
}
}
So in all cases, I think you can see the setting applied? If you aren't seeing what you expect and have a simple example that you can share, that would be helpful.
Hi @Kathleen_DeRusso,
If I do GET _index_template/my_template, here's what I get:
{
"index_templates": [
{
"name": "my_template",
"index_template": {
"index_patterns": [
"my_index_pattern*"
],
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "my_ILM_policy"
},
"number_of_shards": "8",
"number_of_replicas": "1"
}
},
"mappings": {
"_routing": {
"required": false
},
"numeric_detection": false,
"dynamic_date_formats": [
"strict_date_optional_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
],
"dynamic": true,
"_source": {
"excludes": [],
"includes": [],
"enabled": true
},
"date_detection": true
}
},
"composed_of": [],
"data_stream": {
"hidden": false,
"allow_custom_routing": false
}
}
}
]
}
So I was hoping just using this GET that I'd be able to verify the template itself shows the above settings we applied. Am I missing something or doing something wrong?
Can you issue an update call that specifies the max depth settings, before issuing the GET request to update the settings?
Yes, we created the index template, issued the PUT to update the depth limit to 3, and then after the fact I did a GET call to grab the previously mentioned settings.
I can't duplicate this behavior. Can you isolate this behavior to a small reproducable example?
Let me mess around and create a completely new index template. I'll replicate the steps you did a couple comments above and see if I can get it to do either the same behavior we've been seeing, or if hopefully it actually shows all the index template settings from the GET. Thanks!
Hi @Kathleen_DeRusso, so I looked further into this, and noticed something interesting. When using the below PUT that you suggested, it actually created a legacy template instead of updating the composable template I had already created. Is there a PUT _index_template/my_template that I should be using to add the depth limit setting to the composable index template?
PUT _template/my_index_template
{
"index_patterns": [ "index_pattern*" ],
"settings": {
"index.mapping.depth.limit": "3"
}
}
I think this explains why using the GET wasn't giving me the settings we were looking for.
I don't really understand what the problem is here?
Example using component templates:
PUT _component_template/component_template1
{
"template": {
"settings": {
"index.mapping.depth.limit": "3"
}
}
}
PUT _index_template/template_1
{
"index_patterns": ["foo*"],
"composed_of": ["component_template1"]
}
So if I create an index foo-1 and do a subsequent GET call on the index I see that the max depth limit is applied:
GET foo-1
{
"foo-1": {
"aliases": {},
"mappings": {},
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"mapping": {
"depth": {
"limit": "3"
}
},
"number_of_shards": "1",
"provided_name": "foo-1",
"creation_date": "1730235291667",
"number_of_replicas": "1",
"uuid": "MP7wk2MUTWGJ743qQDHUQg",
"version": {
"created": "8100299"
}
}
}
}
}
If I call GET on the index template I see that it's made up of the component templates I specified:
GET _index_template/template_1
{
"index_templates": [
{
"name": "template_1",
"index_template": {
"index_patterns": [
"foo*"
],
"composed_of": [
"component_template1"
]
}
}
]
}
And if I call GET on the component template the setting is there:
GET _component_template/component_template1
{
"component_templates": [
{
"name": "component_template1",
"component_template": {
"template": {
"settings": {
"index": {
"mapping": {
"depth": {
"limit": "3"
}
}
}
}
}
}
}
]
}
This all looks correct and expected to me.
Hi @Kathleen_DeRusso, for clarification, I was referring to the composable index template, not the component index template for applying the index.mapping.depth.limit setting. Is that the only way to apply it the setting to a COMPOSABLE index template? I use the term composable as that is what the modern index templates are referred to now.
Let me start fresh though to explain:
- Create a composable index template in the GUI.
- Using the API, from what you previously mentioned, do
PUT _template/my_template to set index.mapping.depth.limit to 3.
- Doing this specific
PUT does not apply this setting to the just-created composable template, but creates a LEGACY index template with the same name, that DOES show the mapping depth setting.
So my questions are:
- Does a legacy template having the same name as the COMPOSABLE (modern) template work? Are they tied together?
- Per your most recent example, you used a COMPONENT template created through the API, and then applied the COMPONENT template to the COMPOSABLE index template.
I think the weak point for us here is the lack of COMPONENT template understanding, as we don't actively use those in our COMPOSABLE templates. I will read up on those more to gain a better understanding. Thanks!
From our documentation:
Composable templates take precedence over legacy templates. If no composable template matches a given index, a legacy template may still match and be applied.
I've given examples using both the composable templates in the _index_template API and the component templates using the same composable APIs. I don't know how much more help I can be with additional examples here, so I suggest that you review the documentation and experiment. If you find something that you believe is a bug, please don't hesitate to open a Github issue in the Elasticsearch GitHub repo. Thank you!
1 Like