Kibana displays JSON field with "?" in front, but its searchable and acts like seperate fields

[stcdarrell (Darrell Miller)]

i hope this makes sense.. and is in the right group..

i'm using logstash to query a REST API which returns a really large JSON.

That JSON is dumped into a field called "CIF_Source_Body," it works.. within that JSON is a huge amount of nested fields..
This is what kibana displays:
{
"indicator": "125.64.94.211",
"reported_at": "2019-06-27T21:32:20.424476Z",
"timezone": "asia/shanghai",
"latitude": 30.6667,
"peers": [
{
"cc": "CN",
"prefix": "125.64.94.0/24",
"rir": "apnic",
"asn": "4134"
},
{
"cc": "CN",
"prefix": "125.64.94.0/24",
"rir": "apnic",
"asn": "4134"
}
],
"created_at": "2019-06-27T21:32:30.869846",
"description": "identified as sending recursive dns queries to a remote host",
"uuid": "80428a27-de37-4330-98f5-8b1d74a018c5",
"last_at": "2019-06-27T19:32:28.000000Z",
"indicator_ipv4": "125.64.94.211",
"reference": "https://dataplane.org/dnsrd.txt",
"provider": "dataplane.org",
"itype": "ipv4",
"longitude": 104.0667,
"group": "everyone",
"cc": "cn",
"probability": 99.21,
"confidence": 3,
"count": 1,
"asn_desc": "chinanet sichuan telecom internet data center",
"tags": [
"dns",
"scanner"
],
"tlp": "white",
"location": {
"lon": 104.0667,
"lat": 30.6667
},
"region": "sichuan",
"asn": 38283
},
{
"indicator": "125.64.94.211",
"reported_at": "2019-06-26T13:23:54.626049Z",
"timezone": "asia/shanghai",
"latitude": 30.6667,
"peers": [
{
"cc": "CN",
"prefix": "125.64.94.0/24",
"rir": "apnic",
"asn": "4134"
},
{
"cc": "CN",
"prefix": "125.64.94.0/24",
"rir": "apnic",
"asn": "4134"
}
],
"created_at": "2019-06-26T13:24:05.293597",
"description": "identified as sending recursive dns queries to a remote host",
"uuid": "3714c61c-9b11-4693-9e5a-0d166f18afcf",
"last_at": "2019-06-26T10:28:20.000000Z",
"indicator_ipv4": "125.64.94.211",
"reference": "https://dataplane.org/dnsrd.txt",
"provider": "dataplane.org",
"itype": "ipv4",
"longitude": 104.0667,
"group": "everyone",
"cc": "cn",
"probability": 99.03,
"confidence": 3,
"count": 1,
"asn_desc": "chinanet sichuan telecom internet data center",
"tags": [
"dns",
"scanner"
],
"tlp": "white",
"location": {
"lon": 104.0667,
"lat": 30.6667
},
"region": "sichuan",
"asn": 38283
},
{
"indicator": "125.64.94.211",
"reported_at": "2019-06-24T10:02:06.470660Z",
"timezone": "asia/shanghai",
"latitude": 30.6667,
"peers": [
{
"cc": "CN",
"prefix": "125.64.94.0/24",
"rir": "apnic",
"asn": "4134"
},
{
"cc": "CN",
"prefix": "125.64.94.0/24",
"rir": "apnic",
"asn": "4134"
}
],
"created_at": "2019-06-24T10:02:38.556516",
"description": "identified as sending recursive dns queries to a remote host",
"uuid": "d90a79be-ea1f-481c-9b83-7861723ee107",
"last_at": "2019-06-24T08:13:35.000000Z",
"indicator_ipv4": "125.64.94.211",
"reference": "https://dataplane.org/sipquery.txt",
"provider": "dataplane.org",
"itype": "ipv4",
"longitude": 104.0667,
"group": "everyone",
"cc": "cn",
"probability": 99.05,
"confidence": 3,
"count": 1,
"asn_desc": "chinanet sichuan telecom internet data center",
"tags": [
"dns",
"scanner"
],
"tlp": "white",
"location": {
"lon": 104.0667,
"lat": 30.6667
},
"region": "sichuan",
"asn": 38283
}

.. i can search and do visualizations on the fields within this field.. just as i should be able to do in a nested group.. its just not displayed as a nested group..

.. any suggestions on why it isnt displaying within kibana correctly?

thanks

[stcdarrell (Darrell Miller)]

i got it.. not really a kibana issue, more a data issue..
it was an array of jsons.. never seen that before.. a logstash split filter fixes things

i got it.. the results are a JSON array.. so you have to split it up using logstash

 filter {
        if [cif_source_body] {
                split { field => "cif_source_body"}
                split { field => "[cif_source_body][peers]"}
        }


        if [cif_destination_body] {
                split { field => "cif_destination_body"}
                split { field => "[cif_destination_body][peers]"}
        }
}

[rashmi (kulkarni)]

Cool thanks for posting your solution.

Cheers
Rashmi