Hello,
We're on putting a central logging service with many clients and different products.
Depending on source, some standard fields are set at times, at other times not.
We're putting some standard filters to normalize all these common fields.
As simple as : lowercase fields like beat.hostname, beat.name, host, etc.:
mutate {
lowercase => [ "[beat][hostname]", "[beat][name]", "host", ..., ..., ...]
}
Not all sources ship data using beats, not all have all other possible standard fields.
But: when theses fields aren't in the event, it seams to create them empty, because they appear in Kibana with this comment:
This field is present in your elasticsearch mapping but not in any documents in the search results. You may still be able to visualize or search on it.
This can confuse users by adding fields not applicable fields.
I personally doubt, but is it correct behaviour?
Like SQL, if I do upper on null values, it return null...
Better way to manage it than adding a conditional if before each mutate, to check field presence?
Thanks
Bruno