[Syslog output] Erratic message output behavior

TLB35 · January 21, 2026, 10:50am

Hi,

I'm doing this post as I am encountering a "weird" issue with the syslog-output plugin behavior.

My use case is quite simple. My logstash perform a redirection to a nginx cluster. Here is the code snippet :

if [type] == "sentinelone-syslog-udp" {
    syslog {
        host => "vrrp-nginx-back"
        port => "5000"
        protocol => "udp"
        codec => "json"
        rfc => "rfc5424"
    }

However, the behavior of the logstash is weird. Systematically, only one message out of two is correctly sent to the Nginx cluster (I performed a network capture to ensure that the issue was indeed on logstash side and not on the receiver side).
Two noteworthy facts :

if I change the configuration to use the udp output plugin, this behavior disappear, and the message are sent in an even way.
```
udp {
    host => "vrrp-nginx-back"
    port => 5000
    codec => json
}
```
This configuration worked well in the past, with logstash in version 8.17.1. I upgraded my installation to version 9.2.0. Both time, I installed the logstash plugins via the appropriate way described on the documentation (bin/logstash-plugin update).

Here is a summary of the software versions used :

OLD :
logstash 8.17.1
logstash-output-syslog (3.0.5)
logstash-output-udp (3.2.0)

NEW :
logstash-output-syslog (3.1.0)
logstash-output-udp (3.2.0)
logstash 9.2.0

Could this be a software related issue ? Like did the logstash major version update could have “broken” our configuration in some way ?

Thanks in advance for your answers.

Badger · January 21, 2026, 2:01pm

The changes in logstash-output-syslog (3.1.0) were mainly to do with TLS, but also to add rfc5424 support. I suggest removing rfc => "rfc5424" to see if that helps.

TLB35 · January 21, 2026, 2:41pm

Hello,

Thank you for this quick answer. Unfortunately, this change in the configuration did not change anything.

My initial take was that the syslog-output-udp was faulty, but maybe an other piece of software in the logstash stack is at cause.

Badger · January 21, 2026, 2:46pm

Then I would take a look at whether the if [type] == "sentinelone-syslog-udp" { conditional is satisfied for the events that do not get sent.

TLB35 · January 21, 2026, 3:42pm

I tried to comment the if [ type ] to be sure it isn’t the source. I got the exact same output.

To add more information to the post :
My input field looks like this :

    tcp {
        port => 55225
        type => "test-syslog-udp"
    }
    udp {
        port => 55225
        type => "test-syslog-udp"
    }
}

In both scenarios (tcp and udp) it is voluntary to have the type “test-syslog-udp” added to the message.
After that I have a filter phase

    if \[type\] == "test-syslog-udp" {
        mutate {
            add_field => { "client" => "test_logstash" }
            remove_field => \[ "message" \]
        }
    }
}

This phase seems to work fine everytime
And the ouput is

    if \[type\] == "test-syslog-udp" {
        syslog {
            host => "vrrp-nginx-back"
            port => "514"
            protocol => "udp"
            codec => "json"
        }
    }
    file {
        path => "/var/log/logstash/clients/00-test/test_sentinelone.log"
        codec => json_lines
    }
}

The file output helps to be sure that everytime a log is received, logstash processes it to the end

To test the logstash, i use echo "test S1\n" | nc -w 2 localhost 55225

Everytime i test, i got a log output in test_sentinelone.log as follows :

{"@timestamp":"2026-01-21T15:07:33.203420105Z","client":"test_logstash","event":{"original":"test S1\\\\n"},"type":"test-syslog-udp","@version":"1"}
{"@timestamp":"2026-01-21T15:10:09.271211815Z","client":"test_logstash","event":{"original":"test S1\\\\n"},"type":"test-syslog-udp","@version":"1"}

Here i used the command 2 times.

I recorded the outbound traffic of the logstash server and got only one message :
tcpdump port 514 -X

15:10:20.388789 IP XX.XX.XX.XX.37661 > XX.XX.XX.XX: SYSLOG user.notice, length: 190
        0x0000:  4500 00da 3209 4000 4011 c789 0a18 151f  E...2.@.@.......
        0x0010:  0a18 1732 931d 0202 00c6 4158 3c31 333e  ...2......AX<13>
        0x0020:  6a61 6e76 2e20 3231 2031 353a 3130 3a32  janv..21.15:10:2
        0x0030:  3020 257b 686f 7374 7d20 4c4f 4753 5441  0.%{host}.LOGSTA
        0x0040:  5348 5b2d 5d3a 207b 2240 7469 6d65 7374  SH[-]:.{"@timest
        0x0050:  616d 7022 3a22 3230 3236 2d30 312d 3231  amp":"2026-01-21
        0x0060:  5431 353a 3130 3a32 302e 3333 3334 3337  T15:10:20.333437
        0x0070:  3031 365a 222c 2263 6c69 656e 7422 3a22  016Z","client":"
        0x0080:  7465 7374 5f6c 6f67 7374 6173 6822 2c22  test_logstash","
        0x0090:  6576 656e 7422 3a7b 226f 7269 6769 6e61  event":{"origina
        0x00a0:  6c22 3a22 7465 7374 2053 315c 5c6e 227d  l":"test.S1\\n"}
        0x00b0:  2c22 7479 7065 223a 2274 6573 742d 7379  ,"type":"test-sy
        0x00c0:  736c 6f67 2d75 6470 222c 2240 7665 7273  slog-udp","@vers
        0x00d0:  696f 6e22 3a22 3122 7d0a                 ion":"1"}.

At the same time, I recorded the inbound traffic on the receiving nginx cluster and got the same only message :
tcpdump -n port 514 -X

15:10:20.374671 IP XX.XX.XX.XX.37661 > XX.XX.XX.XX.514: SYSLOG user.notice, length: 190
        0x0000:  4500 00da 3209 4000 3f11 c889 0a18 151f  E...2.@.?.......
        0x0010:  0a18 1732 931d 0202 00c6 334e 3c31 333e  ...2......3N<13>
        0x0020:  6a61 6e76 2e20 3231 2031 353a 3130 3a32  janv..21.15:10:2
        0x0030:  3020 257b 686f 7374 7d20 4c4f 4753 5441  0.%{host}.LOGSTA
        0x0040:  5348 5b2d 5d3a 207b 2240 7469 6d65 7374  SH[-]:.{"@timest
        0x0050:  616d 7022 3a22 3230 3236 2d30 312d 3231  amp":"2026-01-21
        0x0060:  5431 353a 3130 3a32 302e 3333 3334 3337  T15:10:20.333437
        0x0070:  3031 365a 222c 2263 6c69 656e 7422 3a22  016Z","client":"
        0x0080:  7465 7374 5f6c 6f67 7374 6173 6822 2c22  test_logstash","
        0x0090:  6576 656e 7422 3a7b 226f 7269 6769 6e61  event":{"origina
        0x00a0:  6c22 3a22 7465 7374 2053 315c 5c6e 227d  l":"test.S1\\n"}
        0x00b0:  2c22 7479 7065 223a 2274 6573 742d 7379  ,"type":"test-sy
        0x00c0:  736c 6f67 2d75 6470 222c 2240 7665 7273  slog-udp","@vers
        0x00d0:  696f 6e22 3a22 3122 7d0a                 ion":"1"}.

Also we are using the pipeline functionality with the following config

pipeline.id: 00-test_pipeline
path.config: "/etc/logstash/conf.d/00-test_pipeline/*.conf"
pipeline.ecs_compatibility: v8
queue.type: persisted
queue.max_bytes: 1900mb
queue.checkpoint.writes: 1
path.queue: "/var/lib/logstash/queue"

Hope this helps to find what causes the issues.

Badger · January 21, 2026, 3:58pm

I cannot think of anything that would cause this.

Rios · January 21, 2026, 6:01pm

Check this, there was issues with persisted queues, maaaaaaybe is related.
Can you temporarly disable queue.type: persisted, and use in memory: queue.type:memory?

Another option is to use the latest LS version 9.2.4, first try .tar without any installation.

Topic		Replies	Views
Logstash not sending syslog to elasticsearch Logstash	42	4282	December 17, 2021
Problem with gathering logs from syslog with logstash Logstash	9	5588	February 7, 2017
Syslog output plugin configuration Logstash	8	8740	July 6, 2017
TCP syslog output not working properly in logstash Logstash	1	1539	December 21, 2017
Cannot make Logstash read syslog Logstash	10	4875	June 6, 2018

[Syslog output] Erratic message output behavior

Related topics