Documentation for dynamic template mappings?

Elastic Stack Elasticsearch
1

Hello,
I have trouble finding API documentation for setting dynamic templates.

I expected this to be documented at Update mapping API | Elasticsearch Guide [8.3] | Elastic , but this just points to "mapping object" (A link to Mapping!) and Mapping parameters. First is a generic page about mappings, second describes only properties of static mappings.

The same problem is with Create or update index template API | Elasticsearch Guide [8.3] | Elastic, this also links just to the "mapping object" (A link to Mapping!), Mapping parameters and Mapping.

Mapping | Elasticsearch Guide [8.3] | Elastic then links to Dynamic mapping and dynamic templates. First describes how dynamic mappings happens by default, second documents only few select examples.

Can somebody please point me to proper documentation of API that sets dynamic mapping templates?
Thank you!

2

If you don't set a mapping then dynamic applies by default.

Or is there something more specific you are looking to do?

3

Thanks for answering this quickly.

Yes, I am specifically looking for a more complete description of this dynamic mapping templates feature.
The best I managed to find is Dynamic templates | Elasticsearch Guide [8.3] | Elastic, but it looks incomplete. For example it claims that Dynamic templates are specified as an array of named objects: like this:

"dynamic_templates": [
    {
      "my_template_name": { 
        ... match conditions ... 
        "mapping": { ... } 
      }
    },

But actually the "mapping" part is not mandatory at all. On the other hand there could be a "runtime" section that is not mentioned in the syntax (and its presence actually forbids "mapping" from being present). I am looking for details like this.

Well, concretely I am trying to ossify/formalize the currently known mappings without losing the agility of a fresh ES installation when it comes to processing of previously unknown fields.
I do not like "dynamic": "enabled", because that does not provide an easy way for kibana users to see that a field is prone to conflicts and/or changes because its type is not fixed.
I also do not like "dynamic":"runtime", because that does not provide adequate performance. The moment the users of the kibana notice the new field, they will expect full performace of it.

A perfect solution would be to have "dynamic": "enabled" but somehow mark the dynamically added fields in a way that alerts users of kibana that this field is not set in stone and may change.

For example I hoped to create a subfield "fieldname.autodetected" and prevent the "fieldname" field from being mapped at all (like Mapping: trying to ignore 'field' but keeping 'field.raw' asks), but found no way to do that. I know I can make the "fieldname" field unindexed, but that will both confuse the users AND occupy two slots out of the 1000-field-limit instead of one.

I could tolerate the "fieldname" field being a runtime field and "fieldname.autodetected" being a normal field, but I do not see any example for this either.

Thus, I was hoping somebody could point me to a more complete documentation where I could find these things.

4
I have tried blocking the original field by a script 
{
	"new dynamic field": {
		"mapping": {
			"type": "boolean",
			"script": {
				"source": "emit(null)"
			},
			"fields": {
				"autotype": {
					"type": "{dynamic_type}"
				}
			},
		"match_mapping_type": "*"
		}
	}
}
  • This does not work, it gives Cannot define multifields on a field with a script
I have tried blocking the field by setting it to boolean + ignore_malformed 
{
	"new dynamic field": {
		"mapping": {
			"type": "boolean",
			"ignore_malformed": "true",
			"fields": {
				"autotype": {
					"type": "{dynamic_type}"
				}
			},
		"match_mapping_type": "*"
		}
	}
}
  • That is documented as unsupported and in practice it gets silently ignored
I have tried blocking the field by setting it to geo_point + ignore_malformed 
{
	"new dynamic field": {
		"mapping": {
			"type": "geo_point",
			"ignore_malformed": "true",
			"fields": {
				"autotype": {
					"type": "{dynamic_type}"
				}
			},
		"match_mapping_type": "*"
		}
	}
}
  • This is documented as supported, but in practice ignore_malformed does not work as expected. Indexed documents are rejected with errors like {"type": "illegal_state_exception", "reason": "found leftover path elements: fieldname."}
I had some success with ignore_maformed + requiring a date prefixed by some klingon 
{
	"new dynamic field": {
		"mapping": {
			"type": "date",
			"format": "'''arlogh Qoylu''''pu''''?'yyyyy.d",
			"ignore_malformed": "true",
			"fields": {
				"autotype": {
					"type": "{dynamic_type}"
				}
			},
		"match_mapping_type": "*"
		}
	}
}
  • This is probably very indefficient, but it does succeed in removing the original field
  • Unfortunately I forgot that kibana hides multi-fields under the main field, which cause the whole field to be marked as "Ignored value" and most of GUI tools to be disabled.
  • But maybe that is a good thing that these unstable fields will show a warning and block filtering by default. That gives me opportunity to brief my colleagues about these fields before I show them the trick that adding the .autotype multi-field into "Selected fields" allows searching as usual.

EDIT1: discover:showMultiFields in Advanced Settings | Kibana Guide [8.3] | Elastic might help with visibility of these multi-fields, but personally I'll just stick to briefing my colleagues manually.

EDIT2: I think this solution triggers a small issue in kibana: When value 13 gets indexed into a field configured to this klingon date type, kibana discover will then show the Ignored value warning with a value of Jan 1, 1970 @00:00:00.013, i.e. interprets the value 13 as unix millisecond timestamp. I think that is a bug. Kibana should show this ignored value as-is, because this conversion brings no benefit and it just makes debugging the ignored value problem unnecessarily harder.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.