# Logstash only receiving DNS log while ignoring DHCP using filebeat
**URL:** https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422
**Category:** Logstash
**Created:** [March 27, 2020, 2:25pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422 "2020-03-27T14:25:43Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@data101](https://discuss.elastic.co/u/data101)
#### Post date: [March 27, 2020, 2:25pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422/1 "2020-03-27T14:25:43Z")
Hello,
I am trying to ship windows server DNS and DHCP logs to my testing ELK environment (Ubuntu). I can see Filebeat is sending the logs but for some reason logstash is only receiving dns logs, not dhcp. I am using latest version of ELK stack.
Filebeat.yml
```
#==== Filebeat inputs =========
filebeat.inputs:
- type: log
enabled: true
paths:
- 'C:\dhcplogs\DhcpSrvLog-*.log'
include_lines: ["^[0-9]"]
document_type: dhcp
fields_under_root: true
close_removed : false
clean_removed : false
ignore_older: 47h
clean_inactive: 48h
fields:
type: dhcp
fields_under_root: true
- type: log
enabled: true
paths:
- 'C:\dnslogs\dns*.log'
include_lines: ["^[0-9]"]
document_type: dns
fields_under_root: true
close_removed : false
clean_removed : false
ignore_older: 47h
clean_inactive: 48h
fields:
type: dns
fields_under_root: true
#===Kibana====================
setup.kibana:
host: "X.X.X.X:5601"
#-------- Logstash output -----------------
output.logstash:
hosts: ["X.X.X.X:5044"]
```
In the filebeat log file, I can see filebeat is sending (or atleast tailing both the logs) :
```
2020-03-27T06:21:05.906-0700 INFO registrar/registrar.go:152 States Loaded from registrar: 10
2020-03-27T06:21:05.907-0700 WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-03-27T06:21:05.907-0700 INFO crawler/crawler.go:72 Loading Inputs: 2
2020-03-27T06:21:05.950-0700 INFO log/input.go:152 Configured paths: [C:\dhcplogs\DhcpSrvLog-*.log]
2020-03-27T06:21:05.950-0700 INFO input/input.go:114 Starting input of type: log; ID: 9252333259376502739
2020-03-27T06:21:05.971-0700 INFO log/harvester.go:297 Harvester started for file: C:\dhcplogs\DhcpSrvLog-Fri.log
2020-03-27T06:21:06.058-0700 INFO log/input.go:152 Configured paths: [C:\dnslogs\dns*.log]
2020-03-27T06:21:06.058-0700 INFO input/input.go:114 Starting input of type: log; ID: 15896225400827294810
2020-03-27T06:21:06.059-0700 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 2
2020-03-27T06:21:06.059-0700 INFO cfgfile/reload.go:171 Config reloader started
2020-03-27T06:21:06.060-0700 INFO cfgfile/reload.go:226 Loading of config files completed.
2020-03-27T06:21:06.080-0700 INFO log/harvester.go:297 Harvester started for file: C:\dnslogs\dns2020-03-26T211811Z.log
2020-03-27T06:21:06.983-0700 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://X.X.X.X:5044))
2020-03-27T06:21:07.001-0700 INFO pipeline/output.go:105 Connection to backoff(async(tcp://X.X.X.X:5044)) established
2020-03-27T06:21:33.649-0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1078,"time":{"ms":375}},"total":{"ticks":2406,"time":{"ms":672},"value":2406},"user":{"ticks":1328,"time":{"ms":297}}},"handles":{"open":256},"info":{"ephemeral_id":"63cee4cc-83f1-4326-a606-07a27cb5cb0d","uptime":{"ms":90150}},"memstats":{"gc_next":14871264,"memory_alloc":11575344,"memory_total":120022320,"rss":3751936},"runtime":{"goroutines":47}},"filebeat":{"events":{"added":3885,"done":3885},"harvester":{"files":{"5e2fdf7a-745f-4e10-a135-6c89cb986b02":{"last_event_published_time":"2020-03-27T06:21:32.034Z","last_event_timestamp":"2020-03-27T06:21:32.034Z","name":"C:\\dhcplogs\\DhcpSrvLog-Fri.log","read_offset":1472824,"size":1471019,"start_time":"2020-03-27T06:21:05.953Z"},"db3c0abb-56e2-49dd-ac7f-197ff93aa4d6":{"last_event_published_time":"2020-03-27T06:21:31.236Z","last_event_timestamp":"2020-03-27T06:21:31.236Z","name":"C:\\dnslogs\\dns2020-03-26T211811Z.log","read_offset":94100204,"size":93977245,"start_time":"2020-03-27T06:21:06.075Z"}},"open_files":2,"running":2,"started":2}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":1958,"batches":5,"total":1958},"read":{"bytes":30},"write":{"bytes":129692}},"pipeline":{"clients":2,"events":{"active":0,"filtered":1927,"published":1958,"retry":1136,"total":3885},"queue":{"acked":1958}}},"registrar":{"states":{"current":10,"update":3885},"writes":{"success":16,"total":16}}}}}
2020-03-27T06:22:03.647-0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1296,"time":{"ms":218}},"total":{"ticks":2702,"time":{"ms":296},"value":2702},"user":{"ticks":1406,"time":{"ms":78}}},"handles":{"open":256},"info":{"ephemeral_id":"63cee4cc-83f1-4326-a606-07a27cb5cb0d","uptime":{"ms":120148}},"memstats":{"gc_next":13915552,"memory_alloc":10364368,"memory_total":133421552,"rss":282624},"runtime":{"goroutines":47}},"filebeat":{"events":{"added":1058,"done":1058},"harvester":{"files":{"5e2fdf7a-745f-4e10-a135-6c89cb986b02":{"last_event_published_time":"2020-03-27T06:21:59.127Z","last_event_timestamp":"2020-03-27T06:21:59.127Z","read_offset":1924,"size":1924},"db3c0abb-56e2-49dd-ac7f-197ff93aa4d6":{"last_event_published_time":"2020-03-27T06:21:56.286Z","last_event_timestamp":"2020-03-27T06:21:56.286Z","read_offset":79912,"size":122959}},"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":537,"batches":5,"total":537},"read":{"bytes":30},"write":{"bytes":38849}},"pipeline":{"clients":2,"events":{"active":0,"filtered":521,"published":537,"total":1058},"queue":{"acked":537}}},"registrar":{"states":{"current":10,"update":1058},"writes":{"success":5,"total":5}}}}}
```
But I can only see DNS logs in Kibana. Here is my filebeat.conf file:
```
input {
beats {
port => 5044
}
}
filter
{
if [type] == "dhcp"
{
dissect {
mapping => {
"message" => "%{ID},%{Date},%{Time},%{Description},%{IP_Address},%{Host_Name},%{MAC_Address},%{User_Name},%{TransactionID},%{QResult},%{Probationtime},%{CorrelationID},%{Dhcid},%{VendorClass_hex},%{VendorClass_ascii},%{UserClass_hex},%{UserClass_ascii},%{RelayAgentInformation},%{DnsRegError}"
}
}
mutate
{
add_field => { "log_timestamp" => "%{Date}-%{Time}" }
}
date {
match => ["log_timestamp", "MM/dd/YY-HH:mm:ss"]
timezone => "America/New_York"
}
if "_dateparsefailure" not in [tags]
{
mutate
{
remove_field=> ['Date', 'Time', 'log_timestamp', 'message']
}
}
}
else if [type] == "dns"
{
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => {
"message" => "%{MS_DNS_DATE:date}\s+%{TIME:time}\s+(?:AM|am|PM|pm)\s+%{DATA:thread_id}\s+%{WORD:dns_type}\s+%{BASE16NUM:packet_id}\s+%{WORD:dns_protocol}\s+%{WORD:dns_direction}\s+%{IP:dns_ip}\s+%{BASE16NUM:xid}\s+%{DATA:response}\s+%{WORD:dns_query_type}\s+\[%{BASE16NUM:hex_flags}\s+%{WORD:rcode_name}\s+%{WORD:Flag}\]\s+%{WORD:query_type_name}\s+%{GREEDYDATA:dns_domain}"
}
}
mutate {
gsub => [
# Remove leading (n)
"dns_domain", "^\(\d+\)", "",
# Remove trailing (n)
"dns_domain", "\(\d+\)$", "",
# Replace inner (n)
"dns_domain", "\(\d+\)", "."
]
}
}
}
output {
if [type] == "dhcp"
{
elasticsearch {
hosts => ["http://localhost:9200"]
index => "dhcp-%{+YYYY.MM.dd}"
}
}
else if [type] == "dns"
{
elasticsearch {
hosts => ["http://localhost:9200"]
index => "dns-%{+YYYY.MM.dd}"
}
}
}
```
where MS\_DNS\_Date is a custom pattern:
```
%{MONTHNUM}/%{MONTHDAY}/%{YEAR}
```
I even tried using grok filter instead of dissect (Tested in grok debugger first) but with same result.
```
if [type] == "dhcp"
{
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => {
"message" => "%{DATA:id},%{MS_DNS_DATE:date},%{TIME:time},%{DATA:Description},%{IPV4:ip},%{DATA:Hostname},%{DATA:MAC_Address},%{DATA:Username},%{GREEDYDATA:Remaining_Log}"
}
}
}
```
I am not sure what to do next. There is nothing in logstash.log file as well. No error nothing. Can anyone help me?
---
### Author:  [@Kim-Kruse-Hansen](https://discuss.elastic.co/u/Kim-Kruse-Hansen)
#### Post date: [March 27, 2020, 2:35pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422/2 "2020-03-27T14:35:26Z")
You have the same type in your filebeat conf.
---
### Author:  [@data101](https://discuss.elastic.co/u/data101)
#### Post date: [March 27, 2020, 2:45pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422/3 "2020-03-27T14:45:23Z")
Thanks for your reply but I am sorry I didn't get you. Can you please elaborate a bit (I am new to ELK)
---
### Author:  [@Kim-Kruse-Hansen](https://discuss.elastic.co/u/Kim-Kruse-Hansen)
#### Post date: [March 27, 2020, 2:53pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422/4 "2020-03-27T14:53:50Z")
If you look in your filebeat.yml.
You define 2 inputs with type: dns
- 'C:\dhcplogs\DhcpSrvLog-\*.log'
include\_lines: ["^[0-9]"]
document\_type: dhcp
fields\_under\_root: true
close\_removed : false
clean\_removed : false
ignore\_older: 47h
clean\_inactive: 48h
fields:
type: dns \<\<\<\< change to dhcp
fields\_under\_root: true
Hope this hels
---
### Author:  [@data101](https://discuss.elastic.co/u/data101)
#### Post date: [March 27, 2020, 2:56pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422/5 "2020-03-27T14:56:38Z")
Oh I am sorry. That was a typo I did while posting here. In actual YML file I have dhcp there. Below is the correct one.
```
paths:
- 'C:\dhcplogs\DhcpSrvLog-*.log'
include_lines: ["^[0-9]"]
document_type: dhcp
close_removed : false
clean_removed : false
ignore_older: 47h
clean_inactive: 48h
fields:
type: dhcp
fields_under_root: true
```
---
### Author:  [@Kim-Kruse-Hansen](https://discuss.elastic.co/u/Kim-Kruse-Hansen)
#### Post date: [March 27, 2020, 3:10pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422/6 "2020-03-27T15:10:29Z")
Simple trick is to dump the output from logstash in a logfile on logstash server.
Add this to your logstash conf output section
```
file {
path => "/tmp/logstah-%{+YYYY.MM.dd.log"
codec => rubydebug { metadata => true }
}
```
Then have a look at the documents in the log file , makes debugging logstash a lot easier. FInd your dhco objects in that and see how it look like
---
### Author:  [@data101](https://discuss.elastic.co/u/data101)
#### Post date: [March 27, 2020, 5:30pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422/7 "2020-03-27T17:30:42Z")
I can see dhcp and dns logs in the log file:
DHCP:
```
{
"agent" => {
"hostname" => "XXXX",
"id" => "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX",
"type" => "filebeat",
"ephemeral_id" => "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX",
"version" => "7.6.1"
},
"Description" => "DNS Update Request",
"QResult" => "6",
"log" => {
"file" => {
"path" => "C:\\dhcplogs\\DhcpSrvLog-Fri.log"
},
"offset" => 2420956
},
"UserClass_hex" => "",
"Probationtime" => "",
"type" => "dhcp",
"IP_Address" => "X.X.X.X",
"TransactionID" => "0",
"ecs" => {
"version" => "1.4.0"
},
"UserClass_ascii" => "",
"host" => {
"hostname" => "XXXXXXXX",
"os" => {
"build" => "17763.379",
"kernel" => "10.0.17763.379 (WinBuild.160101.0800)",
"name" => "Windows Server 2019 Standard",
"family" => "windows",
"version" => "10.0",
"platform" => "windows"
},
"name" => "XXXXXXX",
"id" => "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX",
"architecture" => "x86_64"
},
"@version" => "1",
"ID" => "30",
"RelayAgentInformation" => "",
"User_Name" => "",
"Host_Name" => "XXXXXXXXX",
"VendorClass_hex" => "",
"@metadata" => {
"beat" => "filebeat",
"type" => "_doc",
"version" => "7.6.1",
"ip_address" => "10.220.0.11"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"input" => {
"type" => "log"
},
"@timestamp" => 2020-03-27T14:10:27.000Z,
"Dhcid" => "",
"VendorClass_ascii" => "",
"CorrelationID" => "",
"DnsRegError" => "0",
"MAC_Address" => ""
}
```
DNS:
```
{
"date" => "3/27/2020",
"dns_type" => "PACKET",
"agent" => {
"hostname" => "XXXX",
"id" => "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX",
"type" => "filebeat",
"ephemeral_id" => "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX",
"version" => "7.6.1"
},
"log" => {
"file" => {
"path" => "C:\\dnslogs\\dns2020-03-26T211811Z.log"
},
"offset" => 137354306
},
"dns_direction" => "Rcv",
"type" => "dns",
"thread_id" => "1084",
"xid" => "d2f0",
"ecs" => {
"version" => "1.4.0"
},
"packet_id" => "XXXXXXXXXXXXX",
"dns_ip" => "X.X.X.X",
"host" => {
"hostname" => "XXXXXXXXX",
"os" => {
"build" => "17763.379",
"kernel" => "10.0.17763.379 (WinBuild.160101.0800)",
"name" => "Windows Server 2019 Standard",
"family" => "windows",
"version" => "10.0",
"platform" => "windows"
},
"name" => "XXXXXX",
"id" => "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX",
"architecture" => "x86_64"
},
"hex_flags" => "0001",
"@version" => "1",
"@metadata" => {
"beat" => "filebeat",
"type" => "_doc",
"version" => "7.6.1",
"ip_address" => "10.220.0.11"
},
"message" => "3/27/2020 10:06:34 AM 1084 PACKET XXXXXXXXXXXXXX UDP Rcv 10.220.21.16 d2f0 Q [0001 D NOERROR] A (4)play(6)google(3)com(0)",
"Flag" => "NOERROR",
"rcode_name" => "D",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"input" => {
"type" => "log"
},
"dns_query_type" => "Q",
"@timestamp" => 2020-03-27T17:11:40.824Z,
"query_type_name" => "A",
"dns_domain" => "play.google.com",
"dns_protocol" => "UDP",
"time" => "10:06:34"
}
```
But for some weird reason I am also getting winlogbeat log into this as well. I have winlogbeat set up on different server that sends AD log to Kibana. Its running on port 5050 while filebeat on port 5044. I have followed [https://www.syspanda.com/index.php/2018/05/03/monitoring-active-directory-elk/](https://www.syspanda.com/index.php/2018/05/03/monitoring-active-directory-elk/) link to set up my AD server winlogbeat.
That's the only abnormality I noticed in the log file and failed to understand why winlogbeat data is getting shipped to filebeat and filebeat despite sending the dhcp logs is not visible in kibana. Any kind of help is appreciated
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 24, 2020, 5:30pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422/8 "2020-04-24T17:30:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.