How to create key value from Nested fields

Hi,
I am getting json data in kibana. The data having nested fields like

    aws.datasource.resource.instanceDetails.tags
    {
      "key": "Environment",
      "value": "true"
    },
    {
      "key": "nga",
      "value": "lob"
    },
    {
      "key": "Name",
      "value": "net"
    },
    {
      "key": "llo",
      "value": "UltraRecursive.net"
    },
    {
      "key": "EOP",
      "value": "19.21.65.1"
    }

How to create field as
aws.datasource.resource.instanceDetails.tags : Environment:<value1>;nga:<value2>;Name:<value3> etc.,

Please help me.

You will need to do this in ruby. What structure do you want to end up with? Show it as JSON or rubydebug please.

Thank you for immediate replay. If this work in ruby then provide ruby code or else using kv filter the field can create?
Below is the json format data
{"schemaVersion":"2.0","accountId":"234156789","region":"ca-west-1","partition":"aws","id":"4dfgjkmnby67093sdfvhnm7","arn":"arn:aws:datasource:ca-west-1:234156789:detectortgrfde67aef0db43a6600995eea0fe2/finding/4cb78d76f9fa922fdbe8dc4a765997a9","type":"Recon:EC2/PortProbeUnprotectedPort","resource":{"resourceType":"Instance","instanceDetails":{"instanceId":"i-45tgbfd3","instanceType":"m3.xlarge","launchTime":"2019-09-23T23:20:46Z","platform":null,"productCodes":[],"iamInstanceProfile":null,"networkInterfaces":[],"outpostArn":null,"tags":[{"key":"Environment","value":"true"},{"key":"nga","value":"llo"},{"key":"Name","value":"et"},{"key":"llo","value":"UltraRecursive.net"},{"key":"EOP","value":"19.21.65.1"}],"instanceState":"running","availabilityZone":"ca-west-1b","imageId":"ami-45tgfe34","imageDescription":"null"}},"service":{"serviceName":"datasource","detectorId":"4cb78d76f9fa922fdbe8dc4a765997a9","action":{"actionType":"PORT_PROBE","portProbeAction":{"portProbeDetails":[{"localPortDetails":{"port":80,"portName":"HTTP"},"remoteIpDetails":{"ipAddressV4":"193.80.17.56","organization":{"asn":"45678","asnOrg":"ipinc","isp":"ip inc","org":"ip inc"},"country":{"countryName":"Netherlands"},"city":{"cityName":""},"geoLocation":{"lat":2.3824,"lon":89.95}}}],"blocked":false}},"resourceRole":"TARGET","additionalInfo":{"threatName":"Scanner","threatListName":"ProofPoint"},"evidence":{"threatIntelligenceDetails":[{"threatNames":["Scanner"],"threatListName":"ProofPoint"}]},"eventFirstSeen":"2019-12-18T21:20:17Z","eventLastSeen":"2020-07-21T17:55:27Z","archived":false,"count":15022},"severity":2,"createdAt":"2019-12-18T21:33:19.733Z","updatedAt":"2020-07-21T18:05:18.984Z","title":"Unprotected port on EC2 instance i-45tgbfd3 is being probed.","description":"EC2 instance has an unprotected port which is being probed by a known malicious host."}

The field should be below format
aws.datasource.resource.instanceDetails.tags : Environment:<value1>;nga:<value2>;Name:<value3> etc.,

Do you really want this as one large semicolon separated string or do you want this:


?

I want the field as key+value into key:value separated by semicolon with in one field only.

I'm not at a computer and can't test anything. So my quick suggestion would be to do the same as in the link I had posted and then flatten the hash like this:

But there's probably a prettier solution :slight_smile:

Using ruby code in my logstash for what you suggested in the link

I have tested in my logstash configuration using ruby code what you suggested in link and it gives the output as
headers_transformed.Ansible redirect
headers_transformed.Department WebServices
headers_transformed.Environment Production
headers_transformed.Name PR-Redirect
headers_transformed.aws:autoscaling:groupName PR-Redirect

But I want output as
headers_transformed.tags : Ansible:redirect;Department:WebServices; etc.,

Please help me out for creating the field.

… which is why I had posted the second link. But I'll just try to type a clear solution on my phone: (This is bound to fail, but I hope you'll get the idea even if my code is buggy)

ruby {
  code => '
    tags = {} 
    event.get("[aws][datasource][resource][instancDetails][tags]").each do |item| 
      tags[item["key"]] = item["value"]
    end
    event.set("[aws][datasource][resource][instancDetails][tags]", tags.map{|k,v| "#{k}:#{v}"}.join(";")) 
  '
}

The requirement changed. In message I am getting aws.datasource.resource..instancDetails.tags in this key value pairs are coming.
How can I add key value pairs to another field i.e, tags field for that how write ruby code for that. Already in tags field getting some data(tags=datasource).

Expected output is tags=guardduty;key1:value1;key2:value2 etc,
Please help me out this.

While creating [aws][datasource][resource][instancDetails][tags]=key1:value1;key2:value2; etc., using below code

ruby {
  code => '
    tags = {} 
    event.get("[aws][datasource][resource][instancDetails][tags]").each do |item| 
      tags[item["key"]] = item["value"]
    end
    event.set("[aws][datasource][resource][instancDetails][tags]", tags.map{|k,v| "#{k}:#{v}"}.join(";")) 
  '
}

And I am getting error
Ruby exception occurred: undefined method `each' for nil:NilClass
Please suggest me how resolve this.

Seems like [aws][datasource][resource][instancDetails][tags] doesn't exist, at least for some of your documents. Try to put a condition around the Ruby filter like if [aws][datasource][resource][instancDetails][tags] { … }.

But first make sure that you've got the correct field name because your initial post says that there is a field [aws][datasource][resource][instancDetails][tags], but in your JSON I only see [resource][instancDetails][tags]. Or is that JSON only the content of[aws][datasource]? Then there might just be some events without any tags, but the general structure is correct.

I have given wrong field name in code.Now I changed the field name and it's working fine.

Thank you so much.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.