Logstash.pipeline error in dns plugin crashes logstash


(maxf) #1

Hello. I catch this issue 2nd time: DNS plugin crashes logstash (logstash service stay active, but do nothing untill restart)
In logstash log there is much errors like

[ERROR][logstash.filters.dns ] DNS: timeout on resolving the hostname.

and one last error, that crashes it:

[ERROR][logstash.pipeline ] _dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.block in retriable_request(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:255)", "org.jruby.ext.timeout.Timeout.yieldWithTimeout(org/jruby/ext/timeout/Timeout.java:177)", "org.jruby.ext.timeout.Timeout.timeout(org/jruby/ext/timeout/Timeout.java:117)", "org.jruby.ext.timeout.Timeout$INVOKER$s$timeout.call(org/jruby/ext/timeout/Timeout$INVOKER$s$timeout.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.invokeOther4:timeout(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:254)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.retriable_request(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:254)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.invokeOther2:retriable_request(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:276)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.retriable_getaddress(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:276)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.invokeOther0:retriable_getaddress(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.block in resolve(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.lru_redux_minus_1_dot_1_dot_0.lib.lru_redux.ttl.cache.getset(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/lru_redux-1.1.0/lib/lru_redux/ttl/cache.rb:52)",..........}

Му logstash config:

filter {

if [fields][log_file_type] == "dnslogs"
{
csv
{
columns => ["dns_request_time","source_ip","requested_domain_name","dns_server"]
}
mutate
{
# gsub =>["dns_request_time", """, ""]
gsub =>["source_ip", ",", ""]
# gsub =>["requested_domain_name", """, ""]
# gsub =>["dns_server", """, ""]
# gsub =>["requested_domain_name", """, ""]
gsub =>["requested_domain_name", ".$", ""]
}

mutate 
{ 
add_field => {"source_hostname" => "%{source_ip}"}
#copy => { "source_ip" => "source_hostname" }
}



date
{
	match => ["dns_request_time","M/d/yyyy h:mm:ss a"]
	target => "@timestamp"
}




mutate
{ 
	add_field => {"local" => "false"}
	#add_field => {"subdomain"=> "nulldomain"}
	#add_field => {"sld"=> "nulldomain"}

}
###
grok {
	patterns_dir => ["/etc/logstash/patterns"]
    match => [ "requested_domain_name", "%{HOST:domain}" ]
}
tld
{
source => "domain"
target => "target"
}

if [requested_domain_name] =~ /(?i).*\.XXXX$/
or [requested_domain_name] =~ /(?i)^XXXX\.ru$/
or [requested_domain_name] =~ /(?i).*\.XXXX\.ru$/
or [requested_domain_name] =~ /(?i).*\.XXXX-XXXX\.ru$/
or [requested_domain_name] =~ /(?i).*\.XXXX-XXXX\.ru$/
or [requested_domain_name] =~ /(?i).*\.XXXXXXXX\.ru$/
or [requested_domain_name] =~ /(?i).+\.local$/
or [requested_domain_name] =~ /(?i)^\w+$/
{ 
	mutate { replace => { "local" => "true" }}
}

	dns 
{
failed_cache_size => "2000"
failed_cache_ttl => "600"
hit_cache_size => "2000"
hit_cache_ttl => "600"
max_retries  => "0"
nameserver => [ "192.168.64.253" ]
reverse => [ "source_hostname" ]
action => "replace"
#timeout => 1
}


if [local] != 'true'

{
mutate {
add_field => {"target_host" => "%{requested_domain_name}"}
add_field => {"maliciousIP" => "false"}
add_field => {"maliciousDomain" => "false"}
}

dns 
{
failed_cache_size => "2000"
failed_cache_ttl => "600"
hit_cache_size => "2000"
hit_cache_ttl => "600"
max_retries  => "0"
nameserver => [ "192.168.64.253" ]
resolve => [ "target_host" ]
action => "replace"
#timeout => 0.7
}
translate {
 field => "[target_host]"
 destination => "maliciousIP"
 dictionary_path => '/etc/logstash/conf.d/ipblacklists/alien.yaml'

}
translate {
field => "[requested_domain_name]"
destination => "maliciousDomain"
dictionary_path => '/etc/logstash/conf.d/domainblacklists/malwaredomains.yaml'
}
}

}

}

Whats wrong? logstash 6.2.3


(Ry Biesemeyer) #2

Are there any other log lines? Typically when logstash.pipeline logs an ERROR-level message it includes the text of the exception in addition to the backtrace pasted above.


(maxf) #3

Yah, but message is longer than 7k symbols. Ill split it

[2018-05-02T12:13:50,711][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"target_host", "value"=>["%{requested_domain_name}"]}
[2018-05-02T12:13:50,711][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"maliciousIP", "value"=>["false"]}
[2018-05-02T12:13:50,711][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"maliciousDomain", "value"=>["false"]}
[2018-05-02T12:13:50,713][ERROR][logstash.pipeline ] dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.block in retriable_request(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:255)", "org.jruby.ext.timeout.Timeout.yieldWithTimeout(org/jruby/ext/timeout/Timeout.java:177)", "org.jruby.ext.timeout.Timeout.timeout(org/jruby/ext/timeout/Timeout.java:117)", "org.jruby.ext.timeout.Timeout$INVOKER$s$timeout.call(org/jruby/ext/timeout/Timeout$INVOKER$s$timeout.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.invokeOther4:timeout(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:254)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.retriable_request(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:254)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.invokeOther2:retriable_request(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:276)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.retriable_getaddress(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:276)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.invokeOther0:retriable_getaddress(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.block in resolve(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.lru_redux_minus_1_dot_1_dot_0.lib.lru_redux.ttl.cache.getset(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/lru_redux-1.1.0/lib/lru_redux/ttl/cache.rb:52)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.lru_redux_minus_1_dot_1_dot_0.lib.lru_redux.ttl.cache.RUBY$method$getset$0$VARARGS(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/lru_redux_minus_1_dot_1_dot_0/lib/lru_redux/ttl//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/lru_redux-1.1.0/lib/lru_redux/ttl/cache.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.lru_redux_minus_1_dot_1_dot_0.lib.lru_redux.util.safe_sync.invokeSuper0:-unknown-super-target-(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/lru_redux_minus_1_dot_1_dot_0/lib/lru_redux/util//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/lru_redux-1.1.0/lib/lru_redux/util/safe_sync.rb:26)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.lru_redux_minus_1_dot_1_dot_0.lib.lru_redux.util.safe_sync.block in getset(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/lru_redux-1.1.0/lib/lru_redux/util/safe_sync.rb:26)", "uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.monitor.mon_synchronize(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/monitor.rb:214)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.lru_redux_minus_1_dot_1_dot_0.lib.lru_redux.util.safe_sync.invokeOther2:synchronize(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/lru_redux_minus_1_dot_1_dot_0/lib/lru_redux/util//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/lru_redux-1.1.0/lib/lru_redux/util/safe_sync.rb:25)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.lru_redux_minus_1_dot_1_dot_0.lib.lru_redux.util.safe_sync.getset(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/lru_redux-1.1.0/lib/lru_redux/util/safe_sync.rb:25)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.invokeOther42:getset(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.block in resolve(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:145)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.invokeOther57:each(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:124)",


(maxf) #4

2nd part

"usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.resolve(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:124)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.invokeOther1:resolve(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:97)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_dns_minus_3_dot_0_dot_9.lib.logstash.filters.dns.filter(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.9/lib/logstash/filters/dns.rb:97)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:filter(usr/share/logstash/logstash_minus_core/lib/logstash/filters//usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:do_filter(usr/share/logstash/logstash_minus_core/lib/logstash/filters//usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.block in multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther7:each(usr/share/logstash/logstash_minus_core/lib/logstash/filters//usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", "usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.invokeOther10:multi_filter(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47)", "usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47)", "RUBY.block in initialize((eval):702)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "RUBY.block in initialize((eval):698)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org/jruby/RubyProc$INVOKER$i$0$0$call19.gen)", "RUBY.block in initialize((eval):727)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "RUBY.block in initialize((eval):715)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org/jruby/RubyProc$INVOKER$i$0$0$call19.gen)", "RUBY.block in filter_func((eval):420)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther3:filter_func(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:447)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:447)", "RUBY.worker_loop(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426)", "RUBY.block in start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)", "java.lang.Thread.run(java/lang/Thread.java:748)"], :thread=>"#<Thread:0x1645735f@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[2018-05-02T12:13:50,735][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-05-02T12:13:50,735][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-05-02T12:13:55,737][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-05-02T12:13:55,737][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-05-02T12:14:00,739][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-05-02T12:14:00,739][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-05-02T12:14:05,742][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-05-02T12:14:05,742][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-05-02T12:14:10,743][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-05-02T12:14:10,743][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}


(maxf) #5

Is this what you mean? Have any Idea?


(maxf) #6

the question is still actual


(Ry Biesemeyer) #7

Something is getting truncated from your pasted log message, which prevents me from seeing what the exception is.

The line that begins with:

Should have an exception message next; instead, it jumps straight into the middle of a backtrace.

What I can tell though, is that there is some issue with how Timeouts are being handled.

I can also see that 8 days ago a new version of the plugin was released, which addresses some timeout-related issues and performance issues:

3.0.10

  • Log timeouts as warn instead of error #43
  • Allow concurrent queries when cache enabled #42

-- CHANGELOG.md

You can update to the latest version of the plugin independent of your Logstash version:

bin/logstash-plugin update logstash-filter-dns

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.