Problem with crashing UDP Sender in Logstash

Elastic Stack Logstash
1

Hi,
We use logstash to send UDP data through a secure interface that only allows communication in one direction. It worked very well intil I introduced a little more load to it.

Running ELK stack 6.6.0 with filebeat and auditbeat at the same version.

Now I get this error:

----- ERROR MESSAGE ----- Start
logstash[16381]: [2019-04-30T09:21:57,882][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<Errno::EMSGSIZE: Message too long - No message available>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:438:in send'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-udp-3.0.6/lib/logstash/outputs/udp.rb:24:inblock in register'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-json-3.0.5/lib/logstash/codecs/json.rb:42:in encode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-udp-3.0.6/lib/logstash/outputs/udp.rb:31:inreceive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in block in multi_receive'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:inmulti_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:373:inblock in output_batch'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:372:inoutput_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:324:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:287:inblock in start_workers'"]}
----- ERROR MESSAGE ----- Stop

------------- CONFIGURATION ---------- Stop
We have a configuration that looks like this:

Server1
filebeat/auditbeat -> logstash -> UDP to server 2

Logstash configuration:
input {
beats {
port =>
ssl => true
ssl_certificate => ""
ssl_key => ""
}

}
output {
udp {
host => ""
port =>
codec => "json"
}
}

Server2
UDP from server 1 -> logstash -> elasticsearch

Logstash configuration:
input {
udp {
port =>
codec => "json"
type => "source_udp"
}
}

------------- CONFIGURATION ---------- Start

Any one knows what im doing wrong? Or what to tweak to make it work with more load?

2

Don't use UDP to send data between Logstash hosts. UDP does offer any delivery guarantees and has a limitation on size. Instead use TCP plugins or maybe even a lumberjack output paired with a beats input plugin on the receiving side.

3

Don´t have any choice. We use a data diode (unidirectional) between the servers for protection.

I know its not a good solution but with the data diode I have to use UDP.

For reference. A unit like this one:

4

Then I suspect you may need to guard against messages that are too large and filter these out.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.