Hello,
I'm having problems with Gelf Input plugin dropping messages when listening on UDP.
When a lot of messagess comes in same time, it seems like logstash plugin is dropping them randomly.
E.g. when I restart quarkus pod it only logs out few lines of code, while directly in pod logs I can see all log messages.
Pod logs:
2023-02-10 11:50:28,629 [1] INFO [liq.database] (main) Set default schema name to dbo
2023-02-10 11:50:29,111 [1] INFO [liq.changelog] (main) Reading from DATABASECHANGELOG
2023-02-10 11:50:29,301 [1] INFO [liq.lockservice] (main) Successfully acquired change log lock
2023-02-10 11:50:29,878 [1] INFO [liq.lockservice] (main) Successfully released change log lock
2023-02-10 11:50:30,262 [1] INFO [io.sma.rea.mes.amqp] (main) SRMSG16212: Establishing connection with AMQP broker
2023-02-10 11:50:30,390 [1] INFO [io.quarkus] (main) quarkus on JVM (powered by Quarkus 2.9.1.Final) started in 3.644s. Listening on: http://0.0.0.0:8080
2023-02-10 11:50:30,392 [1] INFO [io.quarkus] (main) Profile prod activated.
2023-02-10 11:50:30,392 [1] INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, hibernate-orm-panache, jdbc-mssql, liquibase, logging-gelf, narayana-jta, rest-client, rest-client-jackson, resteasy-reactive, resteasy-reactive-jackson, scheduler, smallrye-context-propagation, smallrye-health, smallrye-openapi, smallrye-reactive-messaging, smallrye-reactive-messaging-amqp, swagger-ui, vertx]
2023-02-10 11:50:30,539 [1] INFO [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-0) SRMSG16213: Connection with AMQP broker established
Logs written by logstash:
2023-02-10 11:50:29,301 [1] INFO [liq.lockservice] (main) Successfully acquired change log lock
My logstash configuration is:
input {
gelf {
port => 12201
type => "gelf_input"
}
}
output {
if [type] == "gelf_input" {
elasticsearch {
hosts => [ "xxx" ]
user => "username"
password => "password"
}
}
}
When switching to TCP everything works ok, but I would like to avoid using TCP.
What I found related to this problems is this: logstash is not processing most of the gelf messages · Issue #3471 · elastic/logstash · GitHub
Is there any workaround?
Thanks