How to handle nil or null values within mutate add field

[genehunter29009 (gene hunter)]

So I have my data being parsed , transformed correctly however I have this issue where null or nil values cause me problems.

"userActions.cdnBusyTime" => "%{[userActions][cdnBusyTime]}"
output ends up looking like this
"userActions.cdnBusyTime" : "%{[userActions][cdnBusyTime]}",

So far I have tried several things

1. only change if value exists
   if [userActions][cdnResources] {
   mutate {add_field => {"userActions.cdnBusyTime" => "%{[userActions][cdnBusyTime]}"
   }
2. remove value before if null
   filter {
   ruby {
   code => "event.to_hash.delete_if {|field, value| value == '' }"
   }
   }
3. remove theh field later if it contains the name
   if [userActions][cdnBusyTime] in "cdnBusyTime" {
   mutate {
   remove_field => ["[userActions][cdnBusyTime]"]
   }
   }
4. convert and test against !> 0
   mutate {
   convert => [ "[userActions][cdnBusyTime]", "integer" ]
   }
   if [userActions][cdnBusyTime] !> 0 {
   mutate {
   remove_field => [ "[userActions][cdnBusyTime]" ]
   }
   }

There has to be a million ways to do this. But Which one works? any suggestions?

[genehunter29009 (gene hunter)]

I should mention , of course this is nested fields.

[Badger]

@genehunter29009

I should mention , of course this is nested fields.

Well, you think of it as a nested field, but when the substitution fails what is created is a top-level field. So a prune filter with the default blacklist will remove it.

[genehunter29009 (gene hunter)]

thank you for the advice. researching now.

[genehunter29009 (gene hunter)]

So here is my issue I am putting a value in that field regardless of if its null or has a value, so the field will either have a value = 45 or "%{[userActions][cdnBusyTime]}"

I have tried to only add the field if its not null but thats not easy , as you know null does not exist in elasticsearch. So the prune would have to look for this value vs null "%{[userActions][cdnBusyTime]}"

my issue is of course the quotes and % , I dont know how to format that. Where would the actual other code go?

1. put here
   split {
   field => "[userActions]"
   }
2. put here
   mutate {
   add_field => {
   "userActions.name" => "%{[userActions][name]}"
   "userActions.domain" => "%{[userActions][domain]}"
   "userActions.targetUrl" => "%{[userActions][targetUrl]}"
   "userActions.type" => "%{[userActions][type]}"
   "userActions.startTime" => "%{[userActions][startTime]}"
   "userActions.endTime" => "%{[userActions][endTime]}"
   "userActions.duration" => "%{[userActions][duration]}"
   "userActions.application" => "%{[userActions][application]}"
   "userActions.internalApplicationId" => "%{[userActions][internalApplicationId]}"
   "userActions.internalKeyUserActionId" => "%{[userActions][internalKeyUserActionId]}"
   "userActions.speedIndex" => "%{[userActions][speedIndex]}"
   "userActions.errorCount" => "%{[userActions][errorCount]}"
   "userActions.apdexCategory" => "%{[userActions][apdexCategory]}"
   "userActions.networkTime" => "%{[userActions][networkTime]}"
   "userActions.serverTime" => "%{[userActions][serverTime]}"
   "userActions.frontendTime" => "%{[userActions][frontendTime]}"
   "userActions.documentInteractiveTime" => "%{[userActions][documentInteractiveTime]}"
   "userActions.failedImages" => "%{[userActions][failedImages]}"
   "userActions.failedXhrRequests" => "%{[userActions][failedXhrRequests]}"
   "userActions.httpRequestsWithErrors" => "%{[userActions][httpRequestsWithErrors]}"
   "userActions.thirdPartyResources" => "%{[userActions][thirdPartyResources]}"
   "userActions.thirdPartyBusyTime" => "%{[userActions][thirdPartyBusyTime]}"
   "userActions.cdnResources" => "%{[userActions][cdnResources]}"
   "userActions.cdnBusyTime" => "%{[userActions][cdnBusyTime]}"
   "userActions.firstPartyBusyTime" => "%{[userActions][firstPartyBusyTime]}"
   "userActions.domCompleteTime" => "%{[userActions][domCompleteTime]}"
   "userActions.domContentLoadedTime" => "%{[userActions][domContentLoadedTime]}"
   "userActions.loadEventStart" => "%{[userActions][loadEventStart]}"
   "userActions.loadEventEnd" => "%{[userActions][loadEventEnd]}"
   "userActions.navigationStart" => "%{[userActions][navigationStart]}"
   "userActions.requestStart" => "%{[userActions][requestStart]}"
   "userActions.responseStart" => "%{[userActions][responseStart]}"
   "userActions.responseEnd" => "%{[userActions][responseEnd]}"
   "userActions.visuallyCompleteTime" => "%{[userActions][visuallyCompleteTime]}"
   "userActions.keyUserAction" => "%{[userActions][keyUserAction]}"
   }
   remove_field => [ "[userActions]" ]
   }

[Badger]

A prune filter can either whitelist (only retain items in the list) or blacklist (only remove items in the list). It can do this based on either the field name or the field value. I am suggesting that you blacklist based on field values (which can be a regexp).

prune { blacklist_values => [ "%\\{[^}]+\\}" ] }

That regexp is copied from the default value for the blacklist_names option. It is unclear to me why the backslashes are needed (maybe to prevent the value being sprintf'd?), so you might also want to try without them.

[genehunter29009 (gene hunter)]

This is what finally worked for me.

ruby {
code => '
event.to_hash.each { |k, v|
if v.to_s.start_with?("%")
event.remove(k) end }
'
}