Add Nested Field Array Value based on Array String

[Kevin_f]

Hello All,

I would like to have some assistance on how to add a nested field array to a new field based on the key string value.

As you can see in the json output below, I am simply trying to add the nested field [properties][additionalDetails][value]] if "key" = "invitedUserEmailAddress"

I have tried:

filter {
 if "invitedUserEmailAddress" in [properties][additionalDetails] {
    mutate {
      add_field => {  "EmailAddress" => "%{[properties][additionalDetails][][value]}" }
   }
 }
}

Which I am hoping to see the output on kibana as a new field added:

EmailAddress            AanhaneA@net

But can't figure out how to add that paticular value for when only if "key" = "InviteUserEmailAddress"

Any help would be much appreciated. Thanks.

[Jenni]

I think you'll have to use Ruby: Iterate over [properties][additionalDetails] and if the string in key is the one you are looking for, fill the field EmailAddress with value.
It's similar to this: Ruby code for array iteration

[Kevin_f]

Thank you Jenni,

However I am not familiar with ruby code. Not sure how that fits into what I am trying to do.

Something like this?

filter {
 mutate {
      add_field => {  "EmailAddress" => "def" }
  }

ruby {
        code => '
            event.get('[properties][additionalDetails]').each do |item|
  				if item['key'] == 'invitedUserEmailAddress'
    				event.set('EmailAddress', '[properties][additionalDetails][value]')
  				end
			end
            '
        }
}

[Jenni]

That looks nearly right. Use item['value] to access the value within your loop. And wrap your code in double quotes if you use single quotes in the code.

[Kevin_f]

Thanks Jenni,

So like:

filter {
 mutate {
      add_field => {  "EmailAddress" => "def" }
  }

ruby {
        code => "
            event.get("[properties][additionalDetails]").each do |item|
  				if item["key"] == "invitedUserEmailAddress"
    				event.set("EmailAddress", "[properties][additionalDetails][value]")
  				end
			end
            "
        }
}

Isn't item["key"] right?

As shown in the json output in the question, I want to check if the array "key" == "invitedUserEmailAddress"

And if that is true, then add its associated value i.e "value" : "AanhaneA@net" to a new field called EmailAddress

[Badger]

@Kevin_f

	if item["key"] == "invitedUserEmailAddress"
    		event.set("EmailAddress", "[properties][additionalDetails][value]")
  	end

That should be

if item["key"] == "invitedUserEmailAddress"
		event.set("EmailAddress", item["value"])
end

[Jenni]

1. I was talking about the value. You used the correct code to use the key in Ruby. But when you needed the value, you just wrote '[properties][additionalDetails][value]' which would literally just insert the string '[properties][additionalDetails][value]' into the field EmailAddress.
2. If you change the quotes at both places, it won't help at all. The point is that the programming language needs to know which quotes belong to each other. If you write ' to say "This is where my ruby code begins" and then use ' again to say "This is where a string in my ruby code begins", it won't work. What the computer sees is event.get( (the text between the first '...') as the code that you want to execute and a lot of nonsense after that, that doesn't belong there. Then it will cry "syntax error!" and do nothing.

---

ruby {
  code => "
    event.get('[properties][additionalDetails]').each do |item|
      if item['key'] == 'invitedUserEmailAddress'
        event.set('EmailAddress', item['value'])
      end
   end
 "
}

[Kevin_f]

Ahh I see. Sorry I was confused.

Thank you both for this. I shall go away and do these tests. Will come back if I face any issues.
Much appreciated!

[Kevin_f]

Hi,

Seems like this ruby code is invalid according to logstash pipeline.

[FATAL] 2020-06-30 17:11:04.597 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "{", "}" at line 23, column 25 (byte 519) after filter {

The full config file are as follows:

input {
    azureblob {
        storage_account_name => "********"
        storage_access_key => "********"
        container => "*******"
        add_field => { "service" => "azure" }
        tags => [ 'test' ]
        codec => "json"
        registry_lease_duration => 60
        registry_create_policy => resume
        file_head_bytes => 0
        file_tail_bytes => 0
    }
}

filter {
 if [service] == "azure" {
   mutate {
      add_field => {  "EmailAddress" => "def" }
   }
   ruby {
        code => "
             event.get("[properties][additionalDetails]").each do |item|
               if item["key"] == "invitedUserEmailAddress"
                event.set("EmailAddress", item["value"])
               end
             end
        "
   }
 }
}

output {
  if [service] == "azure" {
    elasticsearch {
      hosts => ["https://node1.elastic.test.com:9200"]
      cacert => "/etc/logstash/certs/ca.crt"
      index => "logstash-azure-%{+YYYY.MM}"
      user => "********"
      password => "*******"
    }
  }
}

[Badger]

@Kevin_f

   ruby {
        code => "
             event.get("[properties][additionalDetails]").each do |item|
               if item["key"] == "invitedUserEmailAddress"
                event.set("EmailAddress", item["value"])
               end
             end
        "
   }

Perhaps you did not understand what Jenni was saying. If your code contains double quotes then surround it in single quotes. Change this to

ruby {
    code => '
         event.get("[properties][additionalDetails]").each do |item|
           if item["key"] == "invitedUserEmailAddress"
            event.set("EmailAddress", item["value"])
           end
         end
    '
}

[Kevin_f]

Thank you Badger that worked seemlessly. You're a star!

[Kevin_f]

Hi @Badger/@Jenni

With this ruby code, is there a way to not specify the new field "EmailAddress" if the nested field array value "key" does not equal "invitedUserEmailAddress"

It is because I am seeing this in my index too:

Or instead of seeing the value "def" to at least show "-" or null ?

[Jenni]

You explicitly added the field with that value. I assumed that you did that on purpose. Delete that part of your configuration, if you didn't.

@Kevin_f

 mutate {
      add_field => {  "EmailAddress" => "def" }
  }

[Kevin_f]

Ahhh, sorry I assumed that I had to add a new field first as I did not understand that the ruby code will add that for me

So me removing this "add_field" should fix this issue?

[Jenni]

Yes. event.set() in Ruby will create the field if it does not exist and update it if it does.

[Kevin_f]

Awesome thank you! Much appreciated.