Stopping dynamic mapping of fields from URIs


#1

So we are indexing events where one of their fields is the URI of a web request. This value goes into the field of URI (obviously). However, as you know, the URI can be fairly complicated. With dynamic mapping enabled, when the URI is something like:

search?q=elasticsearch+painless&ie=utf-8&oe=utf-8&client=firefox-b-1

ES will create on field

"URI":"search?q=elasticsearch+painless&ie=utf-8&oe=utf-8&client=firefox-b-1"

and fields like:

"search?q":"elasticsearch+painless&ie=utf-8&oe=utf-8&client=firefox-b-1"

"search?q=elasticsearch+painless&ie":"utf-8&oe=utf-8&client=firefox-b-1"

"utf-8&oe":"utf-8&client=firefox-b-1"

and others. This has happened whether or not I'm using the kv filter in logstash, or GROK or something else.

I have been starting to create mapping for each index and then turning off dynamic mapping for each index to prevent the above. But I also saw this setting:

index.mapping.depth.limit
The maximum depth for a field, which is measured as the number of inner objects. For instance, if all fields are defined at the root object level, then the depth is 1. If there is one object mapping, then the depth is 2, etc. The default is 20.

If I set this to 1, does that mean if I GROK the first field:value making it map at the root level,

ES won't parse any further into the value past the inititial? i.e.

"URI":"search?q=elasticsearch+painless&ie=utf-8&oe=utf-8&client=firefox-b-1"

Thanks....


#2

Turns out the only way to do this is to create a new mapping and turn off dynamic mapping.

and reindex everything prior.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.