How to handle nil or null values within mutate add field

Elastic Stack Logstash
1

So I have my data being parsed , transformed correctly however I have this issue where null or nil values cause me problems.

"userActions.cdnBusyTime" => "%{[userActions][cdnBusyTime]}"
output ends up looking like this
"userActions.cdnBusyTime" : "%{[userActions][cdnBusyTime]}",

So far I have tried several things

  1. only change if value exists
    if [userActions][cdnResources] {
    mutate {add_field => {"userActions.cdnBusyTime" => "%{[userActions][cdnBusyTime]}"
    }
  2. remove value before if null
    filter {
    ruby {
    code => "event.to_hash.delete_if {|field, value| value == '' }"
    }
    }
  3. remove theh field later if it contains the name
    if [userActions][cdnBusyTime] in "cdnBusyTime" {
    mutate {
    remove_field => ["[userActions][cdnBusyTime]"]
    }
    }
  4. convert and test against !> 0
    mutate {
    convert => [ "[userActions][cdnBusyTime]", "integer" ]
    }
    if [userActions][cdnBusyTime] !> 0 {
    mutate {
    remove_field => [ "[userActions][cdnBusyTime]" ]
    }
    }

There has to be a million ways to do this. But Which one works? any suggestions?

2

I should mention , of course this is nested fields.

3

Well, you think of it as a nested field, but when the substitution fails what is created is a top-level field. So a prune filter with the default blacklist will remove it.

4

thank you for the advice. researching now.

5

So here is my issue I am putting a value in that field regardless of if its null or has a value, so the field will either have a value = 45 or "%{[userActions][cdnBusyTime]}"

I have tried to only add the field if its not null but thats not easy , as you know null does not exist in elasticsearch. So the prune would have to look for this value vs null "%{[userActions][cdnBusyTime]}"

my issue is of course the quotes and % , I dont know how to format that. Where would the actual other code go?

  1. put here
    split {
    field => "[userActions]"
    }
  2. put here
    mutate {
    add_field => {
    "userActions.name" => "%{[userActions][name]}"
    "userActions.domain" => "%{[userActions][domain]}"
    "userActions.targetUrl" => "%{[userActions][targetUrl]}"
    "userActions.type" => "%{[userActions][type]}"
    "userActions.startTime" => "%{[userActions][startTime]}"
    "userActions.endTime" => "%{[userActions][endTime]}"
    "userActions.duration" => "%{[userActions][duration]}"
    "userActions.application" => "%{[userActions][application]}"
    "userActions.internalApplicationId" => "%{[userActions][internalApplicationId]}"
    "userActions.internalKeyUserActionId" => "%{[userActions][internalKeyUserActionId]}"
    "userActions.speedIndex" => "%{[userActions][speedIndex]}"
    "userActions.errorCount" => "%{[userActions][errorCount]}"
    "userActions.apdexCategory" => "%{[userActions][apdexCategory]}"
    "userActions.networkTime" => "%{[userActions][networkTime]}"
    "userActions.serverTime" => "%{[userActions][serverTime]}"
    "userActions.frontendTime" => "%{[userActions][frontendTime]}"
    "userActions.documentInteractiveTime" => "%{[userActions][documentInteractiveTime]}"
    "userActions.failedImages" => "%{[userActions][failedImages]}"
    "userActions.failedXhrRequests" => "%{[userActions][failedXhrRequests]}"
    "userActions.httpRequestsWithErrors" => "%{[userActions][httpRequestsWithErrors]}"
    "userActions.thirdPartyResources" => "%{[userActions][thirdPartyResources]}"
    "userActions.thirdPartyBusyTime" => "%{[userActions][thirdPartyBusyTime]}"
    "userActions.cdnResources" => "%{[userActions][cdnResources]}"
    "userActions.cdnBusyTime" => "%{[userActions][cdnBusyTime]}"
    "userActions.firstPartyBusyTime" => "%{[userActions][firstPartyBusyTime]}"
    "userActions.domCompleteTime" => "%{[userActions][domCompleteTime]}"
    "userActions.domContentLoadedTime" => "%{[userActions][domContentLoadedTime]}"
    "userActions.loadEventStart" => "%{[userActions][loadEventStart]}"
    "userActions.loadEventEnd" => "%{[userActions][loadEventEnd]}"
    "userActions.navigationStart" => "%{[userActions][navigationStart]}"
    "userActions.requestStart" => "%{[userActions][requestStart]}"
    "userActions.responseStart" => "%{[userActions][responseStart]}"
    "userActions.responseEnd" => "%{[userActions][responseEnd]}"
    "userActions.visuallyCompleteTime" => "%{[userActions][visuallyCompleteTime]}"
    "userActions.keyUserAction" => "%{[userActions][keyUserAction]}"
    }
    remove_field => [ "[userActions]" ]
    }
6

A prune filter can either whitelist (only retain items in the list) or blacklist (only remove items in the list). It can do this based on either the field name or the field value. I am suggesting that you blacklist based on field values (which can be a regexp).

prune { blacklist_values => [ "%\\{[^}]+\\}" ] }

That regexp is copied from the default value for the blacklist_names option. It is unclear to me why the backslashes are needed (maybe to prevent the value being sprintf'd?), so you might also want to try without them.

7

This is what finally worked for me.

ruby {
code => '
event.to_hash.each { |k, v|
if v.to_s.start_with?("%")
event.remove(k) end }
'
}

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.