Logstash crashes on mutate filter

I'm running logstash 6.1.1 and I started getting a ton of these errors in my logs:

[2019-04-11T09:02:32,320][WARN ][logstash.inputs.udp      ] UDP listener died {:exception=>java.nio.channels.ClosedSelectorException, :backtrace=>["sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)", "org.jruby.util.io.Selector
Pool.put(SelectorPool.java:88)", "org.jruby.util.io.SelectExecutor.selectEnd(SelectExecutor.java:59)", "org.jruby.util.io.SelectExecutor.go(SelectExecutor.java:44)", "org.jruby.RubyIO.select(RubyIO.java:3405)", "org.jruby.RubyIO
$INVOKER$s$0$3$select.call(RubyIO$INVOKER$s$0$3$select.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:743)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)"
, "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)
", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedMode
IRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "o
rg.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(Startu
pInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.int
ernal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:16
1)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEng
ine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:16
5)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(Interpret
erEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBl
ockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyPro
c.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}

I think I narrowed it down to the mutate filter. My conf file is below that I'm testing against:

input {

}

filter {

        if "bro_dns" in [tags] {

                csv {
                        columns => ["ts","uid","sourceAddress","sourcePort","destinationAddress","destinationPort","Protocol","TransactionID","RoundTripTime","Query","QueryClass","QueryClassName","QueryType","QueryTypeName","rcode","rcode_name","AA","TC","RD","RA","Z","Answers","TTLs","Rejected"]
                        separator => "  "
                }
#               mutate {
#                       convert => [ "SourcePort", "integer" ]
#                       convert => [ "DestinationPort", "integer" ]
#                       convert => [ "TransactionID", "integer" ]
#                       convert => [ "QueryClass", "integer" ]
#                       convert => [ "QueryType", "integer" ]
#                       convert => [ "rcode", "integer" ]
#                       lowercase => [ "Query" ]
#               }
                date {
                        match => [ "ts", "UNIX" ]
                }

                if "domain1" in [Query] or "domain2" in [Query] or "in-addr.arpa" in [Query] or "ip6.arpa" in [Query] or "_kerberos" in [Query] { drop {} }

                if[Query]!~"domain1" or [Query]!~"x1" or [Query] != "-" {

                        tld {
                                source => "Query"
                        }
#                        mutate {
#                                rename => { "[tld][domain]" => "highest_registered_domain" }
#                                rename => { "[tld][trd]" => "sub_domain" }
#                                rename => { "[tld][tld]" => "top_level_domain" }
##                               rename => { "[tld][sld]" => "parent_domain" }
#                        }

                }
                if [QueryTypeName] or [Query]!~"highest_registered_domain" or [Query]!~"domain1" and [Query] !~ "Query" and [Query] !~ "ServerName" or [Query] !~ "[\\x00\\x00|\\xe2\\x80]"  {
                        rest {
                                request => {
                                        url => "http://10.1.1.1:8002/cmd=measure/tgt=%{Query}"
                                }
#                               sprintf => true
                                json => false
                                target => "FrequencyScore"
                        }
                }
        }

}

output {

#        stdout {codec => rubydebug}


}

If I comment all the mutate sections out, logstash runs just fine. If I put them in, I get those errors and logstash crashes.

So I'm definitely looking at upgrading, but I just wanted to know if anyone else has seen this.

Craig

OK, this was fixed fairly quickly thanks to @securitymapper (on twitter).
Turns out I had multiple mutate - convert statements in all my conf files and by removing those (which are old and outdated syntax) everywhere, logstash became stable.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.