# How to create key value from Nested fields

**URL:** https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084
**Category:** Logstash
**Created:** [July 21, 2020, 6:45pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084 "2020-07-21T18:45:26Z")
**Posts on this page:** 14
**Page:** 1

<div class="post-metadata">

### Author: ![Ramalakshmi](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)
#### Post date: [July 21, 2020, 6:45pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/1 "2020-07-21T18:45:26Z")

</div>

Hi,  
I am getting json data in kibana. The data having nested fields like

```
    aws.datasource.resource.instanceDetails.tags
    {
      "key": "Environment",
      "value": "true"
    },
    {
      "key": "nga",
      "value": "lob"
    },
    {
      "key": "Name",
      "value": "net"
    },
    {
      "key": "llo",
      "value": "UltraRecursive.net"
    },
    {
      "key": "EOP",
      "value": "19.21.65.1"
    }

```

How to create field as  
`aws.datasource.resource.instanceDetails.tags : Environment:<value1>;nga:<value2>;Name:<value3> etc.,`

Please help me.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 21, 2020, 6:53pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/2 "2020-07-21T18:53:57Z")

</div>

You will need to do this in ruby. What structure do you want to end up with? Show it as JSON or rubydebug please.

---

<div class="post-metadata">

### Author: ![Ramalakshmi](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)
#### Post date: [July 21, 2020, 7:10pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/3 "2020-07-21T19:10:54Z")

</div>

Thank you for immediate replay. If this work in ruby then provide ruby code or else using kv filter the field can create?  
Below is the json format data  
`{"schemaVersion":"2.0","accountId":"234156789","region":"ca-west-1","partition":"aws","id":"4dfgjkmnby67093sdfvhnm7","arn":"arn:aws:datasource:ca-west-1:234156789:detectortgrfde67aef0db43a6600995eea0fe2/finding/4cb78d76f9fa922fdbe8dc4a765997a9","type":"Recon:EC2/PortProbeUnprotectedPort","resource":{"resourceType":"Instance","instanceDetails":{"instanceId":"i-45tgbfd3","instanceType":"m3.xlarge","launchTime":"2019-09-23T23:20:46Z","platform":null,"productCodes":[],"iamInstanceProfile":null,"networkInterfaces":[],"outpostArn":null,"tags":[{"key":"Environment","value":"true"},{"key":"nga","value":"llo"},{"key":"Name","value":"et"},{"key":"llo","value":"UltraRecursive.net"},{"key":"EOP","value":"19.21.65.1"}],"instanceState":"running","availabilityZone":"ca-west-1b","imageId":"ami-45tgfe34","imageDescription":"null"}},"service":{"serviceName":"datasource","detectorId":"4cb78d76f9fa922fdbe8dc4a765997a9","action":{"actionType":"PORT_PROBE","portProbeAction":{"portProbeDetails":[{"localPortDetails":{"port":80,"portName":"HTTP"},"remoteIpDetails":{"ipAddressV4":"193.80.17.56","organization":{"asn":"45678","asnOrg":"ipinc","isp":"ip inc","org":"ip inc"},"country":{"countryName":"Netherlands"},"city":{"cityName":""},"geoLocation":{"lat":2.3824,"lon":89.95}}}],"blocked":false}},"resourceRole":"TARGET","additionalInfo":{"threatName":"Scanner","threatListName":"ProofPoint"},"evidence":{"threatIntelligenceDetails":[{"threatNames":["Scanner"],"threatListName":"ProofPoint"}]},"eventFirstSeen":"2019-12-18T21:20:17Z","eventLastSeen":"2020-07-21T17:55:27Z","archived":false,"count":15022},"severity":2,"createdAt":"2019-12-18T21:33:19.733Z","updatedAt":"2020-07-21T18:05:18.984Z","title":"Unprotected port on EC2 instance i-45tgbfd3 is being probed.","description":"EC2 instance has an unprotected port which is being probed by a known malicious host."}`

The field should be below format  
`aws.datasource.resource.instanceDetails.tags : Environment:<value1>;nga:<value2>;Name:<value3> etc.,`

---

<div class="post-metadata">

### Author: ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [July 21, 2020, 7:16pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/4 "2020-07-21T19:16:01Z")

</div>

Do you really want this as one large semicolon separated string or do you want this:

> [@Split array with key+value fields to key: value - Follow up](https://discuss.elastic.co/t/split-array-with-key-value-fields-to-key-value-follow-up/240469):
>
> Hello, I am trying to convert this input { "request": { "time": "2020-04-30T07:32:13.997Z", "size": "0", "headers": [ { "key": "Correlation-Id", "value": "124f4cdf-6cd7-44ae-a8dc-2ecf2b187f42" }, { "key": "host", "value": "api.local.host" }, { "key": "x-forwarded-for", "value": "1.1.1.1" }, { "key": "x-forwarded-port", "value": "443" …

?

---

<div class="post-metadata">

### Author: ![Ramalakshmi](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)
#### Post date: [July 21, 2020, 7:20pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/5 "2020-07-21T19:20:07Z")

</div>

I want the field as key+value into key:value separated by semicolon with in one field only.

---

<div class="post-metadata">

### Author: ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [July 21, 2020, 7:29pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/6 "2020-07-21T19:29:04Z")

</div>

I'm not at a computer and can't test anything. So my quick suggestion would be to do the same as in the link I had posted and then flatten the hash like this:

> <https://stackoverflow.com/questions/3047007/flattening-hash-into-string-in-ruby>

But there's probably a prettier solution 🙂

---

<div class="post-metadata">

### Author: ![Ramalakshmi](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)
#### Post date: [July 21, 2020, 7:33pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/7 "2020-07-21T19:33:58Z")

</div>

Using ruby code in my logstash for what you suggested in the link

---

<div class="post-metadata">

### Author: ![Ramalakshmi](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)
#### Post date: [July 21, 2020, 7:51pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/8 "2020-07-21T19:51:03Z")

</div>

I have tested in my logstash configuration using ruby code what you suggested in link and it gives the output as  
headers\_transformed.Ansible redirect  
headers\_transformed.Department WebServices  
headers\_transformed.Environment Production  
headers\_transformed.Name PR-Redirect  
headers\_transformed.aws:autoscaling:groupName PR-Redirect

But I want output as  
`headers_transformed.tags : Ansible:redirect;Department:WebServices; etc.,`

Please help me out for creating the field.

---

<div class="post-metadata">

### Author: ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [July 21, 2020, 8:24pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/9 "2020-07-21T20:24:52Z")

</div>

… which is why I had posted the second link. But I'll just try to type a clear solution on my phone: (This is bound to fail, but I hope you'll get the idea even if my code is buggy)

```auto
ruby {
  code => '
    tags = {} 
    event.get("[aws][datasource][resource][instancDetails][tags]").each do |item| 
      tags[item["key"]] = item["value"]
    end
    event.set("[aws][datasource][resource][instancDetails][tags]", tags.map{|k,v| "#{k}:#{v}"}.join(";")) 
  '
}
```

---

<div class="post-metadata">

### Author: ![Ramalakshmi](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)
#### Post date: [July 22, 2020, 6:49pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/10 "2020-07-22T18:49:54Z")

</div>

The requirement changed. In message I am getting aws.datasource.resource..instancDetails.tags in this key value pairs are coming.  
How can I add key value pairs to another field i.e, tags field for that how write ruby code for that. Already in tags field getting some data(tags=datasource).

Expected output is tags=guardduty;key1:value1;key2:value2 etc,  
Please help me out this.

---

<div class="post-metadata">

### Author: ![Ramalakshmi](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)
#### Post date: [July 23, 2020, 3:10pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/13 "2020-07-23T15:10:39Z")

</div>

While creating [aws][datasource][resource][instancDetails][tags]=key1:value1;key2:value2; etc., using below code

```
ruby {
  code => '
    tags = {} 
    event.get("[aws][datasource][resource][instancDetails][tags]").each do |item| 
      tags[item["key"]] = item["value"]
    end
    event.set("[aws][datasource][resource][instancDetails][tags]", tags.map{|k,v| "#{k}:#{v}"}.join(";")) 
  '
}

```

And I am getting error  
Ruby exception occurred: undefined method `each' for nil:NilClass  
Please suggest me how resolve this.

---

<div class="post-metadata">

### Author: ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [July 23, 2020, 3:45pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/14 "2020-07-23T15:45:00Z")

</div>

Seems like `[aws][datasource][resource][instancDetails][tags]` doesn't exist, at least for some of your documents. Try to put a condition around the Ruby filter like `if [aws][datasource][resource][instancDetails][tags] { … }`.

But first make sure that you've got the correct field name because your initial post says that there is a field `[aws][datasource][resource][instancDetails][tags]`, but in your JSON I only see `[resource][instancDetails][tags]`. Or is that JSON only the content of`[aws][datasource]`? Then there might just be some events without any tags, but the general structure is correct.

---

<div class="post-metadata">

### Author: ![Ramalakshmi](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)
#### Post date: [July 23, 2020, 6:52pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/15 "2020-07-23T18:52:30Z")

</div>

I have given wrong field name in code.Now I changed the field name and it's working fine.

Thank you so much.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 20, 2020, 6:52pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084/16 "2020-08-20T18:52:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
