Logstash-forwarder queuing support

(ti log) #1

Hello all,

in https://github.com/elastic/logstash-forwarder/issues/18 we see the issue is closed, therefore I thought, redis support is now included in 0.4.0. But how can redis as target being configured?

The config like this

output {
  redis {
    host => "INDEXER_HOSTNAME"
    data_type => "list"
    key => "logstash"
    codec => json

seems to be only for real logstash not logstash forwarder.

If redis support is still not implemented, can sb please provide examples for another MQ which could buffer the messages ?

Thanks for your help in advance.

(Magnus Bäck) #2

logstash-forwarder does not support Redis. Why do you need to submit messages to Redis? There are probably valid reasons for desiring this but there are also reasons based on misconceptions.

(ti log) #3

Hi Magnus,

I need to submit messages to a queuing service like redis or any other MQ, as sometimes logstash needs to be restarted or I get "Lumberjack input, maximum connection exceeded, new connection are rejected.", which all remain on CLOSE_WAIT!
Currently I run logstash with "-w 12" which seems to resolve the issue but what happens if the process again stops to receive messages from my forwardes?

Best regards,

(Magnus Bäck) #4

logstash-forwarder should back off and retry and the messages should eventually arrive (if not then it's a bug). As long as this overload condition is temporary you should be okay. If Logstash constantly pushes back you have an unstable system and chances are that it would fall over even with a message broker inbetween.

(system) #5