# DNS resolving in JSON document

**URL:** <https://discuss.elastic.co/t/dns-resolving-in-json-document/79016>\
**Category:** Logstash\
**Created:** [March 17, 2017, 12:04pm UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016 "2017-03-17T12:04:58Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 17, 2017, 12:04pm UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/1 "2017-03-17T12:04:58Z")

</div>

I have a nested JSON document which i'm pushing it to Elastic via Logstash.  
To resolve the ip address i have the DNS plugin installed. It works when i format the json as a single row & insert.  
But the time required to convert the json document into rows requires noticeable time which i'm trying to avoid by pushing the full json document into elastic.

My question is how do i resolve & insert the DNS into elastic which is inside the document.

Any help is appreciated.

Thanks,  
Karthick

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 17, 2017, 12:19pm UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/2 "2017-03-17T12:19:47Z")

</div>

What does the document currently look like? What do you want it to look like?

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 17, 2017, 12:31pm UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/3 "2017-03-17T12:31:47Z")

</div>

![](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8f84397b83846d07ecfa2bdda1232773d8d55029.png)[quote="magnusbaeck, post:2, topic:79016, full:true"]  
What does the document currently look like? What do you want it to look like?  
[/quote]

Thanks for the reply.  
Kindly refer the screenshot of my json doc. All i need is to insert this data into elastic along with resolving dns for ip fields.. Suggest me whether inserting this as a document or inserting as rows is the best way..

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 17, 2017, 12:34pm UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/4 "2017-03-17T12:34:50Z")

</div>

Please answer my second question too.

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 20, 2017, 6:00am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/5 "2017-03-20T06:00:10Z")

</div>

![](https://us1.discourse-cdn.com/elastic/original/3X/0/5/053bb3ef722fee634ee268a82599861c43d3c71b.png)[quote="magnusbaeck, post:4, topic:79016, full:true"]  
Please answer my second question too.  
[/quote]

I need an additional field which will store the resolved name of the IP address

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 20, 2017, 6:04am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/6 "2017-03-20T06:04:22Z")

</div>

In addition to this i also want to know how do we insert a json document of size around 100MB into elasticsearch via logstash

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 20, 2017, 6:12am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/7 "2017-03-20T06:12:36Z")

</div>

> I need an additional field which will store the resolved name of the IP address

Normally you'd use a dns filter to resolve the IP address, but looking closer I see that you have an array of entries on multiple levels. Will you ever have more than one item in these arrays? If not you should probably flatten the array. This probably works if you don't flatten the array and if you don't have more than one item in each array:

```plaintext
# The dns filter resolves in place, and since you want
# to keep the source values we copy them to the desired
# destination fields first.
mutate {
  add_field => {
    "[IPV4][dt][0][sip][0][ip-resolv]" => "%{[IPV4][dt][0][sip][0][ip]}"
    "[IPV4][dt][0][sip][0][dip][0][ip-resolv]" => "%{[IPV4][dt][0][sip][0][dip][0][ip]}"
  }
}
dns {
  resolve => [
    "[IPV4][dt][0][sip][0][ip-resolv]",
    "[IPV4][dt][0][sip][0][dip][0][ip-resolv]"
  ]
}

```

> In addition to this i also want to know how do we insert a json document of size around 100MB into elasticsearch via logstash

A 100 MB JSON document, seriously? Have you crammed a large binary into a JSON document or is it a "real" document?

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 20, 2017, 6:16am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/8 "2017-03-20T06:16:29Z")

</div>

Thank you for the reply. Yes the document is a real one. The document contains the network stats for a certain duration which usually ranges from 10MB-100MB..

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 20, 2017, 6:26am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/9 "2017-03-20T06:26:54Z")

</div>

The document contains arrays of multiple levels. The document is a nested JSON.

By flatten the array you mean to convert the document into rows? Am i right?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 20, 2017, 6:36am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/10 "2017-03-20T06:36:02Z")

</div>

> The document contains arrays of multiple levels.

Ouch. Then you have to use a ruby filter to do the resolving.

> By flatten the array you mean to convert the document into rows? Am i right?

There are no rows in JSON so I don't know what you mean. By flatten I mean turn e.g.

```
{"foo": [{"bar": true}]}

```

into

```
{"foo": {"bar": true}}

```

which obviously only works if you only have single-item arrays.

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 20, 2017, 6:37am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/11 "2017-03-20T06:37:51Z")

</div>

Yup that is what i meant..

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 29, 2017, 6:25am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/12 "2017-03-29T06:25:12Z")

</div>

Can you provide me the ruby code to flatten this JSON document?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 29, 2017, 6:27am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/13 "2017-03-29T06:27:14Z")

</div>

If `foo` is a top-level field containing a one-element array of objects this should work:

```
event.set('foo', event.get('foo')[0])
```

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 29, 2017, 6:30am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/14 "2017-03-29T06:30:55Z")

</div>

But my document contains multi-level array.. How do i flatten my nested JSON?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 29, 2017, 6:36am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/15 "2017-03-29T06:36:26Z")

</div>

Same principle as in my example. Start at the top and flatten every level. Using your previous example:

```nohighlight
event.set('dt', event.get('dt')[0])
event.set('[dt][sip]', event.get('[dt][sip]')[0])
event.set('[dt][sip][dip]', event.get('[dt][sip][dip]')[0])

```

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 29, 2017, 6:53am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/16 "2017-03-29T06:53:34Z")

</div>

Hi

```
This is the input file

```

> [  
> {  
> "IPV4": [  
> {  
> "utc": "1483589451",  
> "pbid": "2",  
> "ptid": "0",  
> "pp": "0",  
> "bp": "0",  
> "tp": "598",  
> "tb": "2950",  
> "dt": [  
> {  
> "pcol": "01",  
> "pp": "0",  
> "bp": "0",  
> "tp": "16",  
> "tb": "500",  
> "sip": [  
> {  
> "ip": "1.1.1.1",  
> "pp": "0",  
> "bp": "0",  
> "tp": "16",  
> "tb": "500",  
> "dip": [  
> {  
> "ip": "5.5.5.5",  
> "pp": "0",  
> "bp": "0",  
> "tp": "9",  
> "tb": "200",  
> "spo": [  
> {  
> "pt": "00",  
> "pp": "0",  
> "bp": "0",  
> "tp": "9",  
> "tb": "200",  
> "dpo": [  
> {  
> "pt": "00",  
> "pp": "0",  
> "bp": "0",  
> "tp": "9",  
> "tb": "200"  
> }  
> ]  
> }  
> ]  
> },  
> {  
> "ip": "6.6.6.6",  
> "pp": "0",  
> "bp": "0",  
> "tp": "7",  
> "tb": "300",  
> "spo": [  
> {  
> "pt": "00",  
> "pp": "0",  
> "bp": "0",  
> "tp": "7",  
> "tb": "300",  
> "dpo": [  
> {  
> "pt": "00",  
> "pp": "0",  
> "bp": "0",  
> "tp": "7",  
> "tb": "300"  
> }  
> ]  
> }  
> ]  
> }  
> ]  
> }  
> ]  
> },  
> {  
> "pcol": "02",  
> "pp": "0",  
> "bp": "0",  
> "tp": "5",  
> "tb": "2450",  
> "sip": [  
> {  
> "ip": "2.2.2.2",  
> "pp": "0",  
> "bp": "0",  
> "tp": "1",  
> "tb": "1250",  
> "dip": [  
> {  
> "ip": "7.7.7.7",  
> "pp": "0",  
> "bp": "0",  
> "tp": "1",  
> "tb": "500",  
> "spo": [  
> {  
> "pt": "00",  
> "pp": "0",  
> "bp": "0",  
> "tp": "1",  
> "tb": "500",  
> "dpo": [  
> {  
> "pt": "00",  
> "pp": "0",  
> "bp": "0",  
> "tp": "1",  
> "tb": "500"  
> }  
> ]  
> }  
> ]  
> }  
> ]  
> }  
> ]  
> }  
> ]  
> }  
> ]  
> }  
> ]

This is my desired output json

> {  
> "protocol": "IPV4",  
> "pbid": "2",  
> "utc": "1483589451",  
> "ptid": "0",  
> "pp": "0",  
> "bp": "0",  
> "tp": "22",  
> "tb": "1000",  
> "sip.ip": "1.1.1.1",  
> "sip.pp": "0",  
> "sip.bp": "0",  
> "sip.tp": "17",  
> "sip.tb": "500",  
> "sip.dip.ip": "5.5.5.5",  
> "sip.dip.pp": "0",  
> "sip.dip.bp": "0",  
> "sip.dip.tp": "9",  
> "sip.dip.tb": "200",  
> "sip.dip.spo.pt": "00",  
> "sip.dip.spo.pp": "0",  
> "sip.dip.spo.bp": "0",  
> "sip.dip.spo.tp": "9",  
> "sip.dip.spo.tb": "200",  
> "sip.dip.spo.dpo.pt": "00",  
> "sip.dip.spo.dpo.pp": "0",  
> "sip.dip.spo.dpo.bp": "0",  
> "sip.dip.spo.dpo.tp": "9",  
> "sip.dip.spo.dpo.tb": "200"  
> },  
> {  
> "protocol": "IPV4",  
> "pbid": "2",  
> "utc": "1483589451",  
> "ptid": "0",  
> "pp": "0",  
> "bp": "0",  
> "tp": "22",  
> "tb": "1000",  
> "sip.ip": "1.1.1.1",  
> "sip.pp": "0",  
> "sip.bp": "0",  
> "sip.tp": "17",  
> "sip.tb": "500",  
> "sip.dip.ip": "6.6.6.6",  
> "sip.dip.pp": "0",  
> "sip.dip.bp": "0",  
> "sip.dip.tp": "8",  
> "sip.dip.tb": "300",  
> "sip.dip.spo.pt": "00",  
> "sip.dip.spo.pp": "0",  
> "sip.dip.spo.bp": "0",  
> "sip.dip.spo.tp": "8",  
> "sip.dip.spo.tb": "300",  
> "sip.dip.spo.dpo.pt": "00",  
> "sip.dip.spo.dpo.pp": "0",  
> "sip.dip.spo.dpo.bp": "0",  
> "sip.dip.spo.dpo.tp": "8",  
> "sip.dip.spo.dpo.tb": "300"  
> },  
> {  
> "protocol": "IPV4",  
> "pbid": "2",  
> "utc": "1483589451",  
> "ptid": "0",  
> "pp": "0",  
> "bp": "0",  
> "tp": "22",  
> "tb": "1000",  
> "sip.ip": "2.2.2.2",  
> "sip.pp": "0",  
> "sip.bp": "0",  
> "sip.tp": "5",  
> "sip.tb": "500",  
> "sip.dip.ip": "7.7.7.7",  
> "sip.dip.pp": "0",  
> "sip.dip.bp": "0",  
> "sip.dip.tp": "5",  
> "sip.dip.tb": "500",  
> "sip.dip.spo.pt": "00",  
> "sip.dip.spo.pp": "0",  
> "sip.dip.spo.bp": "0",  
> "sip.dip.spo.tp": "5",  
> "sip.dip.spo.tb": "500",  
> "sip.dip.spo.dpo.pt": "00",  
> "sip.dip.spo.dpo.pp": "0",  
> "sip.dip.spo.dpo.bp": "0",  
> "sip.dip.spo.dpo.tp": "5",  
> "sip.dip.spo.dpo.tb": "500"  
> }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 29, 2017, 7:14am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/17 "2017-03-29T07:14:36Z")

</div>

Sorry, I don't have time to tailor code to your exact specifications.

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 29, 2017, 7:27am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/18 "2017-03-29T07:27:53Z")

</div>

Ok no issues. But i have a few questions:

1. Can this be achieved?
2. Is there any link you can share for reference
3. Would be great if you share me the sample code which matches my requirement

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 29, 2017, 7:35am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/19 "2017-03-29T07:35:38Z")

</div>

> Can this be achieved?

Yes.

> Is there any link you can share for reference

Perhaps [Event API | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-api.html) is helpful. Apart from that it's just normal Ruby.

> Would be great if you share me the sample code which matches my requirement

As I've already said I don't have time to do that.

---

<div class="post-metadata">

**Author:** ![karthickv](https://avatars.discourse-cdn.com/v4/letter/k/a3d4f5/32.png) [@karthickv](https://discuss.elastic.co/u/karthickv)\
**Post date:** [March 29, 2017, 7:37am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016/20 "2017-03-29T07:37:03Z")

</div>

Ok thanks for your time.

[Next page](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016.md?page=2)