DNS request timed out - logstash filter

[Raj_Kumar (Raj)]

Hi All,
Hi there,

This with regards to DNS request time out in logstash filter for cisco firewall logs, after restarting the logstash service it will work for one day and next day i get the logstash stops parsing data due DNS reverse lookup am not sure it could be failed cache or dns request timeout.But logstash service is active and not dead. Error: dns request timed out

lOGSTASH - Jvm options - -Xms40g-Xmx40g

CPU - 32 cores
RAM - 128GB ram

This is my logstash config

input {
tcp {
port => 5544
type => "cisco-fw"
}
udp {
port => 5544
type => "cisco-fw"
workers => 12
}
}


filter {
if [type] == "cisco-fw" {
if [src_ip] {
# mutate {
add_field => { "hostnameSrc" => "%{src_ip}" }
}
dns {
action => "replace"
reverse => [ "hostnameSrc" ]
hit_cache_size => "1000000"
hit_cache_ttl => "300"
failed_cache_size => "10000"
failed_cache_ttl => "10"
periodic_flush => "true"
max_retries => "1"
# nameserver => ["8.8.8.8", "8.8.4.4"]
}
}
if [dst_ip] {
mutate {
add_field => { "hostnameDst" => "%{dst_ip}" }
}
dns {
action => "replace"
reverse => [ "hostnameDst" ]
hit_cache_size => "1000000"
hit_cache_ttl => "300"
failed_cache_size => "10000"
failed_cache_ttl => "10"
periodic_flush => "true"
max_retries => "1"
# nameserver => ["8.8.8.8", "8.8.4.4"]
}
}
}
}

Please do let me know how to fix this issue as its more important for me to DNS reverse lookup.
Sorry if am made some mistakes in config and please do bare it

Thanks,
Raj

[rcowart (Rob Cowart)]

A few things...

1. Logstash doesn't vertically scale very well. If I had those resources I would run multiple individual Logstash instances.

2. When caching is enabled the threadsafe cache class used in the dns plugin will only allow one worker at a time to make a dns query. The result is that event throughput will suffer badly. You will get better performance by disabling caching.

3. DNS latency can also be an issue, so a cache is still recommended. To achieve this, install a local instance of dnsmasq and configure the dns filter to use 127.0.0.1 as a nameserver. You may still have issues with unresolvable reverse lookups. Such NODATA responses are not cached by dnsmasq. But this is still the best method currently.

[Raj_Kumar (Raj)]

Thank you very much I will look in to that