How to create key value from Nested fields

Elastic Stack Logstash
1

Hi,
I am getting json data in kibana. The data having nested fields like

    aws.datasource.resource.instanceDetails.tags
    {
      "key": "Environment",
      "value": "true"
    },
    {
      "key": "nga",
      "value": "lob"
    },
    {
      "key": "Name",
      "value": "net"
    },
    {
      "key": "llo",
      "value": "UltraRecursive.net"
    },
    {
      "key": "EOP",
      "value": "19.21.65.1"
    }

How to create field as
aws.datasource.resource.instanceDetails.tags : Environment:<value1>;nga:<value2>;Name:<value3> etc.,

Please help me.

2

You will need to do this in ruby. What structure do you want to end up with? Show it as JSON or rubydebug please.

3

Thank you for immediate replay. If this work in ruby then provide ruby code or else using kv filter the field can create?
Below is the json format data
{"schemaVersion":"2.0","accountId":"234156789","region":"ca-west-1","partition":"aws","id":"4dfgjkmnby67093sdfvhnm7","arn":"arn:aws:datasource:ca-west-1:234156789:detectortgrfde67aef0db43a6600995eea0fe2/finding/4cb78d76f9fa922fdbe8dc4a765997a9","type":"Recon:EC2/PortProbeUnprotectedPort","resource":{"resourceType":"Instance","instanceDetails":{"instanceId":"i-45tgbfd3","instanceType":"m3.xlarge","launchTime":"2019-09-23T23:20:46Z","platform":null,"productCodes":[],"iamInstanceProfile":null,"networkInterfaces":[],"outpostArn":null,"tags":[{"key":"Environment","value":"true"},{"key":"nga","value":"llo"},{"key":"Name","value":"et"},{"key":"llo","value":"UltraRecursive.net"},{"key":"EOP","value":"19.21.65.1"}],"instanceState":"running","availabilityZone":"ca-west-1b","imageId":"ami-45tgfe34","imageDescription":"null"}},"service":{"serviceName":"datasource","detectorId":"4cb78d76f9fa922fdbe8dc4a765997a9","action":{"actionType":"PORT_PROBE","portProbeAction":{"portProbeDetails":[{"localPortDetails":{"port":80,"portName":"HTTP"},"remoteIpDetails":{"ipAddressV4":"193.80.17.56","organization":{"asn":"45678","asnOrg":"ipinc","isp":"ip inc","org":"ip inc"},"country":{"countryName":"Netherlands"},"city":{"cityName":""},"geoLocation":{"lat":2.3824,"lon":89.95}}}],"blocked":false}},"resourceRole":"TARGET","additionalInfo":{"threatName":"Scanner","threatListName":"ProofPoint"},"evidence":{"threatIntelligenceDetails":[{"threatNames":["Scanner"],"threatListName":"ProofPoint"}]},"eventFirstSeen":"2019-12-18T21:20:17Z","eventLastSeen":"2020-07-21T17:55:27Z","archived":false,"count":15022},"severity":2,"createdAt":"2019-12-18T21:33:19.733Z","updatedAt":"2020-07-21T18:05:18.984Z","title":"Unprotected port on EC2 instance i-45tgbfd3 is being probed.","description":"EC2 instance has an unprotected port which is being probed by a known malicious host."}

The field should be below format
aws.datasource.resource.instanceDetails.tags : Environment:<value1>;nga:<value2>;Name:<value3> etc.,

4

Do you really want this as one large semicolon separated string or do you want this:

?

5

I want the field as key+value into key:value separated by semicolon with in one field only.

6

I'm not at a computer and can't test anything. So my quick suggestion would be to do the same as in the link I had posted and then flatten the hash like this:

But there's probably a prettier solution :slight_smile:

7

Using ruby code in my logstash for what you suggested in the link

8

I have tested in my logstash configuration using ruby code what you suggested in link and it gives the output as
headers_transformed.Ansible redirect
headers_transformed.Department WebServices
headers_transformed.Environment Production
headers_transformed.Name PR-Redirect
headers_transformed.aws:autoscaling:groupName PR-Redirect

But I want output as
headers_transformed.tags : Ansible:redirect;Department:WebServices; etc.,

Please help me out for creating the field.

9

… which is why I had posted the second link. But I'll just try to type a clear solution on my phone: (This is bound to fail, but I hope you'll get the idea even if my code is buggy)

ruby {
  code => '
    tags = {} 
    event.get("[aws][datasource][resource][instancDetails][tags]").each do |item| 
      tags[item["key"]] = item["value"]
    end
    event.set("[aws][datasource][resource][instancDetails][tags]", tags.map{|k,v| "#{k}:#{v}"}.join(";")) 
  '
}
10

The requirement changed. In message I am getting aws.datasource.resource..instancDetails.tags in this key value pairs are coming.
How can I add key value pairs to another field i.e, tags field for that how write ruby code for that. Already in tags field getting some data(tags=datasource).

Expected output is tags=guardduty;key1:value1;key2:value2 etc,
Please help me out this.

13

While creating [aws][datasource][resource][instancDetails][tags]=key1:value1;key2:value2; etc., using below code

ruby {
  code => '
    tags = {} 
    event.get("[aws][datasource][resource][instancDetails][tags]").each do |item| 
      tags[item["key"]] = item["value"]
    end
    event.set("[aws][datasource][resource][instancDetails][tags]", tags.map{|k,v| "#{k}:#{v}"}.join(";")) 
  '
}

And I am getting error
Ruby exception occurred: undefined method `each' for nil:NilClass
Please suggest me how resolve this.

14

Seems like [aws][datasource][resource][instancDetails][tags] doesn't exist, at least for some of your documents. Try to put a condition around the Ruby filter like if [aws][datasource][resource][instancDetails][tags] { … }.

But first make sure that you've got the correct field name because your initial post says that there is a field [aws][datasource][resource][instancDetails][tags], but in your JSON I only see [resource][instancDetails][tags]. Or is that JSON only the content of[aws][datasource]? Then there might just be some events without any tags, but the general structure is correct.

15

I have given wrong field name in code.Now I changed the field name and it's working fine.

Thank you so much.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.