10-20% message loss when using Stomp Output

[Mr.EoniX (ia@lego.com)]

Hi there,

We are experiencing 10-20% message drop when sending messages from Logstash 5.2 to a Universal Messaging Queue with the STOMP protocol. (Also verified with Logstash 2.4)

Config to reproduce:

input {
        tcp {
                port => 5001
                type => "umrelay"
        }
}

output {
        file {
            path => "/tmp/stash.out"
        }
        if [type] == "umrelay" {
            stomp {
                 destination => "/test/testqueue"
                 host => "x.x.x.x"
                port => 17092
            }
        }
        stdout{}
}

We are using python to send 100.000 messages to the Logstash TCP input - and outs the messages to a file, stomp and console.

The file contains all 100.000 messages
The console writes all 100.000 messages
The Stomp output produces 83.560 messages

The loss happens at rates above 100 pr. second.

We have verified the STOMP functionality on UM by running 100.000 messages to it directly with Python - and all messages are persisted.
We have also tested multiple input path to logstash but we consistantly get everything written to file-out and console, but 10-20% is lost on the STOMP output.

Operating System:
Source: Debian
Target: OpenSuse

Sample Data:
The message sent from Python in all experiements is "i\n" where i is the current message number in an iteration of 100.000

Steps to reproduce:
Send messages to any logstash input at a message rate above 100 pr. second, and output on Stomp Output.
Count your input and output and assess the message drop. We see about 10-20% drop.

This is a problem - And would like to see a fix to this

Best regards,

Mr.EoniX