Packetbeat-DNS index and template correction

Elastic Stack Beats
1

HI all, I am struggling to get the correct information into Elasticsearch. I am using BIND DNS server running Packetbeats direct output to logstash, thereafter to elasticsearch and Kibana. The two fields that are causing some challenge are: dns.additionals and dns.authorities, the output look as follows respectively:
dns.additionals {
"class": "512",
"data": "",
"name": "",
"ttl": 32768,
"type": "OPT"
}
dns.answers {
"class": "IN",
"data": "23.214.151.174",
"name": "e1706.g.akamaiedge.net",
"ttl": 19,
"type": "A"
}
So it looks like there is a filed not split however not sure where the challenge is, in the logstash filter or the elasticsearch index template? Here is the configuration for logstash:
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

Any assistance will be truly appreciated.

2

Here is an sample of a capture:
{"@timestamp":"2016-06-01T18:29:57.514Z","beat":{"hostname":"BIND","name":"BIND"},"bytes_in":64,"bytes_out":222,"client_ip":"192.168.88.250","client_port":56458,"client_proc":"","client_server":"","count":1,"direction":"out","dns":{"additionals":[{"class":"512","data":"","name":"","ttl":32768,"type":"OPT"}],"additionals_count":1,"answers":[{"class":"IN","data":"geover-prod.dodsp.mp.microsoft.com.nsatc.net","name":"geover-prod.do.dsp.mp.microsoft.com","ttl":3360,"type":"CNAME"},{"class":"IN","data":"prod.do.dsp.mp.microsoft.com.edgekey.net","name":"geover-prod.dodsp.mp.microsoft.com.nsatc.net","ttl":299,"type":"CNAME"},{"class":"IN","data":"e1706.g.akamaiedge.net","name":"prod.do.dsp.mp.microsoft.com.edgekey.net","ttl":685,"type":"CNAME"},{"class":"IN","data":"23.214.151.174","name":"e1706.g.akamaiedge.net","ttl":19,"type":"A"}],"answers_count":4,"authorities_count":0,"flags":{"authoritative":false,"recursion_allowed":true,"recursion_desired":true,"truncated_response":false},"id":41071,"op_code":"QUERY","question":{"class":"IN","name":"geover-prod.do.dsp.mp.microsoft.com","type":"A"},"response_code":"NOERROR"},"ip":"8.8.8.8","method":"QUERY","port":53,"proc":"","query":"class IN, type A, geover-prod.do.dsp.mp.microsoft.com","resource":"geover-prod.do.dsp.mp.microsoft.com","responsetime":292,"server":"","status":"OK","transport":"udp","type":"dns"}

3

Did you install the Elasticsearch index template provided with Packetbeat? See https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-template.html

Is your elasticsearch output in Logstash setup to write to the correct index with the correct type? See https://www.elastic.co/guide/en/beats/libbeat/current/logstash-installation.html#logstash-setup

4

I have followed these guides and also removed the filter in logstash, however the sections are still not sorted in different fields. It looks like all information between the [] brackets are not filtered into separate fields e.g. [{"class":"512","data":"","name":"","ttl":32768,"type":"OPT"}]. in some of the responses there are also more than one response, is it possible to tag these items with e.g.
currently:
dns.additionals {
"class": "512",
"data": "",
"name": "",
"ttl": 32768,
"type": "OPT"
}

desired:
dns.additionals.class 512
dns.additionals.data
dns.additionals.name
dns.additionals.ttl 32768
dns.additionals.type OPT

5

Another example for the answer:
dns.answers {
"class": "IN",
"data": "217.69.139.201",
"name": "mail.ru",
"ttl": 47,
"type": "A"
},
{
"class": "IN",
"data": "94.100.180.200",
"name": "mail.ru",
"ttl": 47,
"type": "A"
},
{
"class": "IN",
"data": "94.100.180.202",
"name": "mail.ru",
"ttl": 47,
"type": "A"
},
{
"class": "IN",
"data": "217.69.139.202",
"name": "mail.ru",
"ttl": 47,
"type": "A"
}

so the same approach as above however to have the information in separate fields and to also be able to tag the dns.answers.data 217.69.139.201 with geoip information for each answar

6

The dns.answers and dns.additions fields are arrays because the DNS message allows there to be multiple responses.

There's not really any good way to flatten the array and store it (in what key would you place the data?). It sounds like the question that should be asked is how can Logstash be used to enrich an array of objects containing IP addresses with GeoIP information.

7

Thank you very much for the feedback and suggestion, I have opened a request on the logstash section:

8

This topic was automatically closed after 21 days. New replies are no longer allowed.