Logstash crashes on mutate filter

Elastic Stack Logstash
1

I'm running logstash 6.1.1 and I started getting a ton of these errors in my logs:

[2019-04-11T09:02:32,320][WARN ][logstash.inputs.udp      ] UDP listener died {:exception=>java.nio.channels.ClosedSelectorException, :backtrace=>["sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)", "org.jruby.util.io.Selector
Pool.put(SelectorPool.java:88)", "org.jruby.util.io.SelectExecutor.selectEnd(SelectExecutor.java:59)", "org.jruby.util.io.SelectExecutor.go(SelectExecutor.java:44)", "org.jruby.RubyIO.select(RubyIO.java:3405)", "org.jruby.RubyIO
$INVOKER$s$0$3$select.call(RubyIO$INVOKER$s$0$3$select.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:743)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)"
, "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)
", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedMode
IRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "o
rg.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(Startu
pInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.int
ernal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:16
1)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEng
ine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:16
5)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(Interpret
erEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBl
ockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyPro
c.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}

I think I narrowed it down to the mutate filter. My conf file is below that I'm testing against:

input {

}

filter {

        if "bro_dns" in [tags] {

                csv {
                        columns => ["ts","uid","sourceAddress","sourcePort","destinationAddress","destinationPort","Protocol","TransactionID","RoundTripTime","Query","QueryClass","QueryClassName","QueryType","QueryTypeName","rcode","rcode_name","AA","TC","RD","RA","Z","Answers","TTLs","Rejected"]
                        separator => "  "
                }
#               mutate {
#                       convert => [ "SourcePort", "integer" ]
#                       convert => [ "DestinationPort", "integer" ]
#                       convert => [ "TransactionID", "integer" ]
#                       convert => [ "QueryClass", "integer" ]
#                       convert => [ "QueryType", "integer" ]
#                       convert => [ "rcode", "integer" ]
#                       lowercase => [ "Query" ]
#               }
                date {
                        match => [ "ts", "UNIX" ]
                }

                if "domain1" in [Query] or "domain2" in [Query] or "in-addr.arpa" in [Query] or "ip6.arpa" in [Query] or "_kerberos" in [Query] { drop {} }

                if[Query]!~"domain1" or [Query]!~"x1" or [Query] != "-" {

                        tld {
                                source => "Query"
                        }
#                        mutate {
#                                rename => { "[tld][domain]" => "highest_registered_domain" }
#                                rename => { "[tld][trd]" => "sub_domain" }
#                                rename => { "[tld][tld]" => "top_level_domain" }
##                               rename => { "[tld][sld]" => "parent_domain" }
#                        }

                }
                if [QueryTypeName] or [Query]!~"highest_registered_domain" or [Query]!~"domain1" and [Query] !~ "Query" and [Query] !~ "ServerName" or [Query] !~ "[\\x00\\x00|\\xe2\\x80]"  {
                        rest {
                                request => {
                                        url => "http://10.1.1.1:8002/cmd=measure/tgt=%{Query}"
                                }
#                               sprintf => true
                                json => false
                                target => "FrequencyScore"
                        }
                }
        }

}

output {

#        stdout {codec => rubydebug}


}

If I comment all the mutate sections out, logstash runs just fine. If I put them in, I get those errors and logstash crashes.

So I'm definitely looking at upgrading, but I just wanted to know if anyone else has seen this.

Craig

2

OK, this was fixed fairly quickly thanks to @securitymapper (on twitter).
Turns out I had multiple mutate - convert statements in all my conf files and by removing those (which are old and outdated syntax) everywhere, logstash became stable.