DNS resolving in JSON document

I have a nested JSON document which i'm pushing it to Elastic via Logstash.
To resolve the ip address i have the DNS plugin installed. It works when i format the json as a single row & insert.
But the time required to convert the json document into rows requires noticeable time which i'm trying to avoid by pushing the full json document into elastic.

My question is how do i resolve & insert the DNS into elastic which is inside the document.

Any help is appreciated.

Thanks,
Karthick

What does the document currently look like? What do you want it to look like?

[quote="magnusbaeck, post:2, topic:79016, full:true"]
What does the document currently look like? What do you want it to look like?
[/quote]

Thanks for the reply.
Kindly refer the screenshot of my json doc. All i need is to insert this data into elastic along with resolving dns for ip fields.. Suggest me whether inserting this as a document or inserting as rows is the best way..

Please answer my second question too.

[quote="magnusbaeck, post:4, topic:79016, full:true"]
Please answer my second question too.
[/quote]

I need an additional field which will store the resolved name of the IP address

In addition to this i also want to know how do we insert a json document of size around 100MB into elasticsearch via logstash

I need an additional field which will store the resolved name of the IP address

Normally you'd use a dns filter to resolve the IP address, but looking closer I see that you have an array of entries on multiple levels. Will you ever have more than one item in these arrays? If not you should probably flatten the array. This probably works if you don't flatten the array and if you don't have more than one item in each array:

# The dns filter resolves in place, and since you want
# to keep the source values we copy them to the desired
# destination fields first.
mutate {
  add_field => {
    "[IPV4][dt][0][sip][0][ip-resolv]" => "%{[IPV4][dt][0][sip][0][ip]}"
    "[IPV4][dt][0][sip][0][dip][0][ip-resolv]" => "%{[IPV4][dt][0][sip][0][dip][0][ip]}"
  }
}
dns {
  resolve => [
    "[IPV4][dt][0][sip][0][ip-resolv]",
    "[IPV4][dt][0][sip][0][dip][0][ip-resolv]"
  ]
}

In addition to this i also want to know how do we insert a json document of size around 100MB into elasticsearch via logstash

A 100 MB JSON document, seriously? Have you crammed a large binary into a JSON document or is it a "real" document?

Thank you for the reply. Yes the document is a real one. The document contains the network stats for a certain duration which usually ranges from 10MB-100MB..

The document contains arrays of multiple levels. The document is a nested JSON.

By flatten the array you mean to convert the document into rows? Am i right?

The document contains arrays of multiple levels.

Ouch. Then you have to use a ruby filter to do the resolving.

By flatten the array you mean to convert the document into rows? Am i right?

There are no rows in JSON so I don't know what you mean. By flatten I mean turn e.g.

{"foo": [ {"bar": true}]}

into

{"foo": {"bar": true}}

which obviously only works if you only have single-item arrays.

Yup that is what i meant..

Can you provide me the ruby code to flatten this JSON document?

If foo is a top-level field containing a one-element array of objects this should work:

event.set('foo', event.get('foo')[0])

But my document contains multi-level array.. How do i flatten my nested JSON?

Same principle as in my example. Start at the top and flatten every level. Using your previous example:

event.set('dt', event.get('dt')[0])
event.set('[dt][sip]', event.get('[dt][sip]')[0])
event.set('[dt][sip][dip]', event.get('[dt][sip][dip]')[0])

Hi

This is the input file

[
{
"IPV4": [
{
"utc": "1483589451",
"pbid": "2",
"ptid": "0",
"pp": "0",
"bp": "0",
"tp": "598",
"tb": "2950",
"dt": [
{
"pcol": "01",
"pp": "0",
"bp": "0",
"tp": "16",
"tb": "500",
"sip": [
{
"ip": "1.1.1.1",
"pp": "0",
"bp": "0",
"tp": "16",
"tb": "500",
"dip": [
{
"ip": "5.5.5.5",
"pp": "0",
"bp": "0",
"tp": "9",
"tb": "200",
"spo": [
{
"pt": "00",
"pp": "0",
"bp": "0",
"tp": "9",
"tb": "200",
"dpo": [
{
"pt": "00",
"pp": "0",
"bp": "0",
"tp": "9",
"tb": "200"
}
]
}
]
},
{
"ip": "6.6.6.6",
"pp": "0",
"bp": "0",
"tp": "7",
"tb": "300",
"spo": [
{
"pt": "00",
"pp": "0",
"bp": "0",
"tp": "7",
"tb": "300",
"dpo": [
{
"pt": "00",
"pp": "0",
"bp": "0",
"tp": "7",
"tb": "300"
}
]
}
]
}
]
}
]
},
{
"pcol": "02",
"pp": "0",
"bp": "0",
"tp": "5",
"tb": "2450",
"sip": [
{
"ip": "2.2.2.2",
"pp": "0",
"bp": "0",
"tp": "1",
"tb": "1250",
"dip": [
{
"ip": "7.7.7.7",
"pp": "0",
"bp": "0",
"tp": "1",
"tb": "500",
"spo": [
{
"pt": "00",
"pp": "0",
"bp": "0",
"tp": "1",
"tb": "500",
"dpo": [
{
"pt": "00",
"pp": "0",
"bp": "0",
"tp": "1",
"tb": "500"
}
]
}
]
}
]
}
]
}
]
}
]
}
]

This is my desired output json

{
"protocol": "IPV4",
"pbid": "2",
"utc": "1483589451",
"ptid": "0",
"pp": "0",
"bp": "0",
"tp": "22",
"tb": "1000",
"sip.ip": "1.1.1.1",
"sip.pp": "0",
"sip.bp": "0",
"sip.tp": "17",
"sip.tb": "500",
"sip.dip.ip": "5.5.5.5",
"sip.dip.pp": "0",
"sip.dip.bp": "0",
"sip.dip.tp": "9",
"sip.dip.tb": "200",
"sip.dip.spo.pt": "00",
"sip.dip.spo.pp": "0",
"sip.dip.spo.bp": "0",
"sip.dip.spo.tp": "9",
"sip.dip.spo.tb": "200",
"sip.dip.spo.dpo.pt": "00",
"sip.dip.spo.dpo.pp": "0",
"sip.dip.spo.dpo.bp": "0",
"sip.dip.spo.dpo.tp": "9",
"sip.dip.spo.dpo.tb": "200"
},
{
"protocol": "IPV4",
"pbid": "2",
"utc": "1483589451",
"ptid": "0",
"pp": "0",
"bp": "0",
"tp": "22",
"tb": "1000",
"sip.ip": "1.1.1.1",
"sip.pp": "0",
"sip.bp": "0",
"sip.tp": "17",
"sip.tb": "500",
"sip.dip.ip": "6.6.6.6",
"sip.dip.pp": "0",
"sip.dip.bp": "0",
"sip.dip.tp": "8",
"sip.dip.tb": "300",
"sip.dip.spo.pt": "00",
"sip.dip.spo.pp": "0",
"sip.dip.spo.bp": "0",
"sip.dip.spo.tp": "8",
"sip.dip.spo.tb": "300",
"sip.dip.spo.dpo.pt": "00",
"sip.dip.spo.dpo.pp": "0",
"sip.dip.spo.dpo.bp": "0",
"sip.dip.spo.dpo.tp": "8",
"sip.dip.spo.dpo.tb": "300"
},
{
"protocol": "IPV4",
"pbid": "2",
"utc": "1483589451",
"ptid": "0",
"pp": "0",
"bp": "0",
"tp": "22",
"tb": "1000",
"sip.ip": "2.2.2.2",
"sip.pp": "0",
"sip.bp": "0",
"sip.tp": "5",
"sip.tb": "500",
"sip.dip.ip": "7.7.7.7",
"sip.dip.pp": "0",
"sip.dip.bp": "0",
"sip.dip.tp": "5",
"sip.dip.tb": "500",
"sip.dip.spo.pt": "00",
"sip.dip.spo.pp": "0",
"sip.dip.spo.bp": "0",
"sip.dip.spo.tp": "5",
"sip.dip.spo.tb": "500",
"sip.dip.spo.dpo.pt": "00",
"sip.dip.spo.dpo.pp": "0",
"sip.dip.spo.dpo.bp": "0",
"sip.dip.spo.dpo.tp": "5",
"sip.dip.spo.dpo.tb": "500"
}

Sorry, I don't have time to tailor code to your exact specifications.

Ok no issues. But i have a few questions:

  1. Can this be achieved?
  2. Is there any link you can share for reference
  3. Would be great if you share me the sample code which matches my requirement

Can this be achieved?

Yes.

Is there any link you can share for reference

Perhaps https://www.elastic.co/guide/en/logstash/current/event-api.html is helpful. Apart from that it's just normal Ruby.

Would be great if you share me the sample code which matches my requirement

As I've already said I don't have time to do that.

Ok thanks for your time.