DNS Filter Lookup Failures since 6.7 upgrade


(Andy Millett) #1

Hi Guys,

Hoping someone can offer some guidance. Since our LS upgraded to 6.7.0, we've consistently seen issues with LS resolving DNS entries which appears to build over time. E.g -

[logstash.filters.dns ] DNS: timeout on resolving address. {:field=>"[destination-address]", :value=>"x.x.x.x"}

It's normal for us to see 'some' addresses which are unresolvable, but what appears to be happening is over time (say 1-2 hours) LS will log more and more problems until eventually it stops processing ingested messages causing ES/Kibana dashboards to miss data.

If we restart the LS service, it's fine again, but only for a few hours. It doesn't matter whether it's the middle of peak period, or in the middle of the night.

As I said, before the upgrade it was fine, and locally on the LS server, we can run nslookup/dig to resolve addresses just-fine.

We're ingesting logs from various sources - Bro IDS and Blue Coat via rsyslog or filebeat, along with Juniper, Meru, Pulse Secure logs via Syslog

Any ideas? Anyone else having a similar issue?

There's no error logged for this by LS. It 'just' stops processing until restart. With Metricbeat and Filebeat, thats fine, but with syslog, we end up missing data.

Cheers
Andy


(Andy Millett) #2

Hi All,

For evidence -

[2019-03-29T21:51:34,061][WARN ][logstash.filters.dns     ] DNS: timeout on resolving address. 
{:field=>"[destination-address]", :value=>"10.250.37.62"}
root@logstash00:/var/log/logstash# nslookup 10.250.37.62
Server:		10.250.91.67
Address:	10.250.91.67#53

62.37.250.10.in-addr.arpa	name = sar-xxxx.xxxxx.

Andy


(jahlives) #3

We run into the same problem after updating to ELK 6.7
About one hour after restarting logstash we get the same error messages for (it looks like) every logline processed by logstash. So we're seeing quite huge delays. The only thing that works so far is to disable dns ptr lookups completly in our filters.
After restarting logstash all runs smooth for a while. We first though that the issue might be this https://github.com/logstash-plugins/logstash-filter-dns/issues/40 but it seems our resolv.rb is already correctly patched


(piton) #4

Got the same issue after 6.7.0 update.
6.7.1 update didn't fix the issue.

[2019-04-17T10:18:20,265][WARN ][logstash.filters.dns ] DNS: timeout on resolving address. {:field=>"host", :value=>"172.23.6.9"}

root@syslog-core01:/var/log# host 172.23.6.9
9.6.23.172.in-addr.arpa domain name pointer host2.core.example.com.

logstash-filter-dns plugin version: 3.0.12