# DNS Filter Lookup Failures since 6.7 upgrade **URL:** \ **Category:** Logstash\ **Created:** [March 29, 2019, 4:30pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577 "2019-03-29T16:30:52Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![millap](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@millap](https://discuss.elastic.co/u/millap)\ **Post date:** [March 29, 2019, 4:30pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/1 "2019-03-29T16:30:52Z")
Hi Guys, Hoping someone can offer some guidance. Since our LS upgraded to 6.7.0, we've consistently seen issues with LS resolving DNS entries which appears to build over time. E.g - [logstash.filters.dns] DNS: timeout on resolving address. {:field=\>"[destination-address]", :value=\>"x.x.x.x"} It's normal for us to see 'some' addresses which are unresolvable, but what appears to be happening is over time (say 1-2 hours) LS will log more and more problems until eventually it stops processing ingested messages causing ES/Kibana dashboards to miss data. If we restart the LS service, it's fine again, but only for a few hours. It doesn't matter whether it's the middle of peak period, or in the middle of the night. As I said, before the upgrade it was fine, and locally on the LS server, we can run nslookup/dig to resolve addresses just-fine. We're ingesting logs from various sources - Bro IDS and Blue Coat via rsyslog or filebeat, along with Juniper, Meru, Pulse Secure logs via Syslog Any ideas? Anyone else having a similar issue? There's no error logged for this by LS. It 'just' stops processing until restart. With Metricbeat and Filebeat, thats fine, but with syslog, we end up missing data. Cheers Andy ---
**Author:** ![millap](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@millap](https://discuss.elastic.co/u/millap)\ **Post date:** [March 29, 2019, 9:57pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/2 "2019-03-29T21:57:40Z")
Hi All, For evidence - ``` [2019-03-29T21:51:34,061][WARN][logstash.filters.dns] DNS: timeout on resolving address. {:field=>"[destination-address]", :value=>"10.250.37.62"} root@logstash00:/var/log/logstash# nslookup 10.250.37.62 Server: 10.250.91.67 Address: 10.250.91.67#53 62.37.250.10.in-addr.arpa name = sar-xxxx.xxxxx. ``` Andy ---
**Author:** ![jahlives](https://avatars.discourse-cdn.com/v4/letter/j/73ab20/32.png) [@jahlives](https://discuss.elastic.co/u/jahlives)\ **Post date:** [April 10, 2019, 1:18pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/3 "2019-04-10T13:18:42Z")
We run into the same problem after updating to ELK 6.7 About one hour after restarting logstash we get the same error messages for (it looks like) every logline processed by logstash. So we're seeing quite huge delays. The only thing that works so far is to disable dns ptr lookups completly in our filters. After restarting logstash all runs smooth for a while. We first though that the issue might be this [https://github.com/logstash-plugins/logstash-filter-dns/issues/40](https://github.com/logstash-plugins/logstash-filter-dns/issues/40) but it seems our resolv.rb is already correctly patched ---
**Author:** ![piton](https://avatars.discourse-cdn.com/v4/letter/p/6bbea6/32.png) [@piton](https://discuss.elastic.co/u/piton)\ **Post date:** [April 17, 2019, 10:23am UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/4 "2019-04-17T10:23:02Z")
Got the same issue after 6.7.0 update. 6.7.1 update didn't fix the issue. [2019-04-17T10:18:20,265][WARN][logstash.filters.dns] DNS: timeout on resolving address. {:field=\>"host", :value=\>"172.23.6.9"} root@syslog-core01:/var/log# host 172.23.6.9 9.6.23.172.in-addr.arpa domain name pointer [host2.core.example.com](http://host2.core.example.com). logstash-filter-dns plugin version: 3.0.12 ---
**Author:** ![piton](https://avatars.discourse-cdn.com/v4/letter/p/6bbea6/32.png) [@piton](https://discuss.elastic.co/u/piton)\ **Post date:** [April 22, 2019, 4:06pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/5 "2019-04-22T16:06:15Z")
Even 7.0.0 update didn't fix it. Any suggestions/idea? ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [April 25, 2019, 3:56am UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/6 "2019-04-25T03:56:41Z")
Heads up to say that I believe I just found the cause of this regression and it relates to a library update to the resolver code. I will be following up shortly but for now the only workaround is to downgrade to the latest 6.6 series (6.6.2 as of now). Just downgrading the dns filter plugin version will not help. ---
**Author:** ![christian\_ha](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@christian\_ha](https://discuss.elastic.co/u/christian_ha)\ **Post date:** [April 25, 2019, 6:28am UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/7 "2019-04-25T06:28:55Z")
Same Problem here with LS Version 6.7.1. We planed a elastic upgrade to 7.0 in production today but after this problem we would not do that. ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [April 25, 2019, 3:34pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/8 "2019-04-25T15:34:06Z")
Yes, this affects all versions starting at and after v6.7, including v7 and the only workaround for now to downgrade to v6. Progress can be tracked in [https://github.com/logstash-plugins/logstash-filter-dns/issues/51](https://github.com/logstash-plugins/logstash-filter-dns/issues/51) ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [April 25, 2019, 6:39pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/9 "2019-04-25T18:39:35Z")
I may have a temporary workaround I'd like to validate, any feedback appreciated. If you are using the `nameserver` option of the dns filter **with multiple** hosts configured OR you are not using the `nameserver` option but have multiple servers configured in `/etc/resolv.conf` then as a **tentative temporary** workaround you can try to use a **SINGLE** server in either the `nameserver` option or in your `/etc/resolv.conf` and see if it solves. ---
**Author:** ![millap](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@millap](https://discuss.elastic.co/u/millap)\ **Post date:** [April 25, 2019, 7:37pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/10 "2019-04-25T19:37:00Z")
Hi @colinsurprenant, Thanks for the updates, and confirmation there's a problem to fix. I've applied your logic to our test cluster, and am waiting for some log entries. We have multiple entires inn resolv.conf, so I've switched the LS conf files to use only one on nameserver. I have had some, but no-where near as many so far as before. Probably needs a couple of hours though. One oddity is this, however - ``` [2019-04-25T20:19:43,370][WARN][logstash.filters.dns] DNS: timeout on resolving address. {:field=>"[destination-address]", :value=>"209.112.114.33"} root@elk00:~# nslookup > 209.112.114.33 ;; Truncated, retrying in TCP mode. 33.114.112.209.in-addr.arpa name = k4.nstld.com. 33.114.112.209.in-addr.arpa name = l4.nstld.com. 33.114.112.209.in-addr.arpa name = a22.verisigndns.com. 33.114.112.209.in-addr.arpa name = f4.nstld.com. 33.114.112.209.in-addr.arpa name = ns2.euro909.com. 33.114.112.209.in-addr.arpa name = a23.verisigndns.com. 33.114.112.209.in-addr.arpa name = ns0.netnames.net. 33.114.112.209.in-addr.arpa name = ns1.netnames.net. 33.114.112.209.in-addr.arpa name = ns1.ascio.net. 33.114.112.209.in-addr.arpa name = ns2.domainnetwork.se. 33.114.112.209.in-addr.arpa name = ns3.ascio.net. 33.114.112.209.in-addr.arpa name = ns2.dnsvisa.com. 33.114.112.209.in-addr.arpa name = g4.nstld.com. 33.114.112.209.in-addr.arpa name = a21.verisigndns.com. 33.114.112.209.in-addr.arpa name = ns2.webipdns.com.au. 33.114.112.209.in-addr.arpa name = ns3.netnames.net. 33.114.112.209.in-addr.arpa name = ns5.netnames.net. 33.114.112.209.in-addr.arpa name = a2.verisigndns.com. 33.114.112.209.in-addr.arpa name = pdns1.cscdns.net. 33.114.112.209.in-addr.arpa name = indom30.indomco.fr. 33.114.112.209.in-addr.arpa name = ns7.netnames.net. 33.114.112.209.in-addr.arpa name = indom10.indomco.com. 33.114.112.209.in-addr.arpa name = dns1.cscdns.net. 33.114.112.209.in-addr.arpa name = indom130.indomco.org. ``` Not sure if the switch to TCP might have caused the filter a problem? Andy ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [April 25, 2019, 7:48pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/11 "2019-04-25T19:48:02Z")
Thanks @millap for the followup. The timeout on `209.112.114.33` seems unrelated. When this bug is triggered, most if not all requests will timeout, occasional timeout is _normal_. ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [April 25, 2019, 9:31pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/12 "2019-04-25T21:31:04Z")
Incoming fix in [https://github.com/logstash-plugins/logstash-filter-dns/pull/52](https://github.com/logstash-plugins/logstash-filter-dns/pull/52) ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [April 29, 2019, 7:55pm UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/13 "2019-04-29T19:55:29Z")
The dns filter v3.0.13 has just been published to address this issue. You can update this plugins using ```auto bin/logstash-plugin update logstash-filter-dns ``` Please let us know how this new version is working for you. Colin ---
**Author:** ![christian\_ha](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@christian\_ha](https://discuss.elastic.co/u/christian_ha)\ **Post date:** [May 2, 2019, 5:57am UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/14 "2019-05-02T05:57:44Z")
After updating to v3.0.13 on 30th of april the problem did not occur anymore. I am going to monitor this further. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 30, 2019, 5:57am UTC](https://discuss.elastic.co/t/dns-filter-lookup-failures-since-6-7-upgrade/174577/15 "2019-05-30T05:57:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.