Logstash-forwarder queuing support

Elastic Stack Logstash
1

Hello all,

in https://github.com/elastic/logstash-forwarder/issues/18 we see the issue is closed, therefore I thought, redis support is now included in 0.4.0. But how can redis as target being configured?

The config like this

output {
  redis {
    host => "INDEXER_HOSTNAME"
    data_type => "list"
    key => "logstash"
    codec => json
  }

seems to be only for real logstash not logstash forwarder.

If redis support is still not implemented, can sb please provide examples for another MQ which could buffer the messages ?

Thanks for your help in advance.
T.

2

logstash-forwarder does not support Redis. Why do you need to submit messages to Redis? There are probably valid reasons for desiring this but there are also reasons based on misconceptions.

3

Hi Magnus,

I need to submit messages to a queuing service like redis or any other MQ, as sometimes logstash needs to be restarted or I get "Lumberjack input, maximum connection exceeded, new connection are rejected.", which all remain on CLOSE_WAIT!
Currently I run logstash with "-w 12" which seems to resolve the issue but what happens if the process again stops to receive messages from my forwardes?

Best regards,
T

4

logstash-forwarder should back off and retry and the messages should eventually arrive (if not then it's a bug). As long as this overload condition is temporary you should be okay. If Logstash constantly pushes back you have an unstable system and chances are that it would fall over even with a message broker inbetween.