Dynamically adding fields with ES field mapping template enabled

[krishna_chaitanya]

I am unable to decide between dynamic mapping and giving a template before hand.

I have configured logstash with file input and ES output.

Say, ES output plugin is configured with field mapping template. Currently, I have defined 20 fields and their data types in the template. But, in the future, if I want to add new fields on the fly into my log files, is it possible? Do I have to edit my mapping ? or is Logstash able to recognize these newly fields datatypes through dynamic template?

The problem I have is with nested types. I have doc like this:

{
field1 : value1,
field2:  [ {... }, {...}]
}

So, field2 is essentially a nested type, which if not defined in template (dynamic template enabled), is just like an object with no datatype. In Kibana, I am unable to access this field2 from Kibana scripted field.

doc['field2'].value throws me this error: No field found for [field2] in mapping with types [].
If I map it to nested_type using pre-defined template, then I could access this field, but I do not know if this new template will not allow me to add new fields in the future to my documents.

Any help appreciated !!

[magnusbaeck (Magnus Bäck)]

This is more of an Elasticsearch question so you'll probably get better answers in that group.

Having an index template and allowing dynamic mapping isn't contradictory. Unless you explicitly disable dynamic mapping, new fields can be added without changes to the template and the index's mappings.

[krishna_chaitanya]

Thanks @magnusbaeck That's exactly what I wanted.