# DNS slower with cache enabled?

**URL:** <https://discuss.elastic.co/t/dns-slower-with-cache-enabled/96744>\
**Category:** Logstash\
**Created:** [August 11, 2017, 10:02am UTC](https://discuss.elastic.co/t/dns-slower-with-cache-enabled/96744 "2017-08-11T10:02:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tom\_Mortimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_mortimer/32/21049_2.png) [@Tom\_Mortimer](https://discuss.elastic.co/u/Tom_Mortimer)\
**Post date:** [August 11, 2017, 10:02am UTC](https://discuss.elastic.co/t/dns-slower-with-cache-enabled/96744/1 "2017-08-11T10:02:03Z")

</div>

Hi,

I'm working on a logstash installation which feeds log messages from various sources (syslog, Windows, Checkpoint firewalls etc.) into Elasticsearch for analysis and monitoring. Everything was working nicely until I added a DNS filter to resolve domain names to IP addresses and vice versa:

```
...
if [SI_Source_IP] =~ /^\d+\.\d+\.\d+\.\d+/ {
  dns {
    reverse => ["SI_Source_FQDN"]
    action => "replace"
  }
}
else {
  dns {
    resolve => ["SI_Source_IP"]
    action => "replace"
  }
}
...

```

(SI\_Source\_FQDN and SI\_Source\_IP contain the same value before this fragment executes, either a FQDN or an IP address). Similar code deals with the Destination\_IP/FQDN.

I've got four filter workers and four output workers (any more than this didn't seem to increase performance.)

Without any DNS lookup I was getting a throughput of 4000 messages/s. With the DNS code added this drops to around 1000/s. But what really surprised me was that adding caching _reduces_ the DNS performance, to about 600 messages/s:

```
...
hit_cache_size => 8000
hit_cache_ttl => 300
failed_cache_size => 1000
failed_cache_ttl => 10
...

```

Logstash is running on two VMs, each with two virtual CPU cores and 4GB RAM. According to the customer they are "well resourced" and the current load is not topping out the CPUs.

Am I making any stupid mistakes? Is there anything I could do to improve performance? Why is adding caching making things worse rather than better?

cheers,  
Tom

---

<div class="post-metadata">

**Author:** ![Tom\_Mortimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_mortimer/32/21049_2.png) [@Tom\_Mortimer](https://discuss.elastic.co/u/Tom_Mortimer)\
**Post date:** [August 14, 2017, 3:26pm UTC](https://discuss.elastic.co/t/dns-slower-with-cache-enabled/96744/2 "2017-08-14T15:26:15Z")

</div>

I've just noticed that the threadsafe cache class used in the plugin will only allow one worker at a time to call out to DNS (unless my understanding of Ruby synchronisation is way off.) So I'm going to fork the plugin and modify it to allow concurrent lookups.

---

<div class="post-metadata">

**Author:** ![Tom\_Mortimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_mortimer/32/21049_2.png) [@Tom\_Mortimer](https://discuss.elastic.co/u/Tom_Mortimer)\
**Post date:** [August 17, 2017, 3:50pm UTC](https://discuss.elastic.co/t/dns-slower-with-cache-enabled/96744/3 "2017-08-17T15:50:46Z")

</div>

Solution here -

> **[Better performance with the Logstash DNS filter -](http://www.flax.co.uk/blog/2017/08/17/better-performance-logstash-dns-filter/)**
>
> We’ve been working on a project for a customer which uses Logstash to read messages from Kafka and write them to Elasticsearch. It also parses the messages into fields, and depending on the content type does DNS lookups (both forward and … More

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2017, 3:51pm UTC](https://discuss.elastic.co/t/dns-slower-with-cache-enabled/96744/4 "2017-09-14T15:51:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.