Udp input with multiline codec - Exception in inputworker

Elastic Stack Logstash
1

Logstash falls every 2-3 days. Is there anything i can do about this?

"@timestamp"=>2018-01-10T16:30:00.523Z, "message"=>"", "tags"=>["_dissectfailure"]}}
[2018-01-10T16:30:02,636][WARN ][org.logstash.dissect.Dissector] Dissector mapping, field found in event but it was empty {"field"=>"message", "event"=>{"@version"=>"1", "host"=>"172.31.21.189", "@timestamp"=>2018-01-10T16:30:02.631Z, "message"=>"", "tags"=>["_dissectfailure"]}}
[2018-01-10T16:30:26,740][WARN ][org.logstash.dissect.Dissector] Dissector mapping, field found in event but it was empty {"field"=>"message", "event"=>{"@version"=>"1", "host"=>"172.31.21.189", "@timestamp"=>2018-01-10T16:30:26.734Z, "message"=>"", "tags"=>["_dissectfailure"]}}
[2018-01-10T16:30:42,918][ERROR][logstash.inputs.udp      ] Exception in inputworker {"exception"=>#<ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users>, "backtrace"=>["org/jruby/RubyArray.java:1256:in `<<'", "org/jruby/RubyArray.java:1271:in `push'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.7/lib/logstash/codecs/multiline.rb:211:in `buffer'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.7/lib/logstash/codecs/multiline.rb:269:in `do_previous'", "org/jruby/RubyMethod.java:119:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.7/lib/logstash/codecs/multiline.rb:205:in `block in decode'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.7/lib/logstash/codecs/multiline.rb:198:in `decode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.1.2/lib/logstash/inputs/udp.rb:118:in `inputworker'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.1.2/lib/logstash/inputs/udp.rb:89:in `block in udp_listener'"]}

[2018-01-10T16:30:42,919][ERROR][logstash.inputs.udp      ] Exception in inputworker {"exception"=>#<ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users>, "backtrace"=>["org/jruby/RubyArray.java:1256:in `<<'", "org/jruby/RubyArray.java:1271:in `push'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.7/lib/logstash/codecs/multiline.rb:211:in `buffer'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.7/lib/logstash/codecs/multiline.rb:269:in `do_previous'", "org/jruby/RubyMethod.java:119:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.7/lib/logstash/codecs/multiline.rb:205:in `block in decode'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.7/lib/logstash/codecs/multiline.rb:198:in `decode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.1.2/lib/logstash/inputs/udp.rb:118:in `inputworker'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.1.2/lib/logstash/inputs/udp.rb:89:in `block in udp_listener'"]}

config:

input {
  udp {
    port => 5200
    codec => multiline {
      pattern => "^([a-zA-Z0-9-]+.[a-zA-Z0-9-]+.[a-zA-Z0-9-]+) (DEBUG .+)|(INFO .+)|(ERROR .+)|(WARNING .+)"
      negate => true
      what => "previous"
    }
  }
}

filter {
    dissect {
      mapping => { "message" => "%{host} %{levelname} %{name} %{asctime} %{+asctime} %{module} %{process} %{thread} %{message}"}
    }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }

}

Logstash 6.0.0 Centos Linux 3.10.0-693.5.2.el7.x86_64
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

2

As delivery of UDP message are not necessarily ordered nor guaranteed, I am not sure how well a multiline codec is going to work. What is the rationale behind using it here? Why not use TCP instead?

3

multiline basically works fine. The problem was when collecting logs from several hosts, for logs from one server, the host field was contain ip from second server, i was resolve this problem with the host field added to msg before transfer. UDP is faster.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.