Rsyslog to Redis to logstash to elasticsearch without logstash forwarder?

Elastic Stack Logstash
1

I see lots of tutorials like this one: http://www.everybodyhertz.co.uk/setting-up-a-relk-stack-a-how-to/
suggesting that you use logstash forwarder to send logs to redis before sending it to logstash and then to elasticsearch. I'd like to know if it is possible to send from rsyslog/syslog to redis directly without logstash forwarder then from redis to logstash and then to elasticsearch.
What I'd like to do:
rsyslog->redis->logstash-indexer->elasticsearch

What other people are doing:
rsyslog->logstash-forwarder->redis->logstash-indexer->elasticsearch

The reason I'd like to send it directly to redis is I'd like to be able to cache data in a way so that if logstash crashed then I would still have the data sent from rsyslog to logstash in that short while that logstash was down.

Also is redis the right choice for this. Would it be possible to use varnish for this purpose instead?

Regards.

2

kurtcoke,

Good question. I am looking to do a similar setup - Pipe syslog into Redis

I ran across this tool https://github.com/genebean/sawyer earlier today but not sure if it will work with rsyslog. I am going to try it with syslog and NXLog.

1 Like
3

This is really more of a rsyslog question. I am not aware of rsyslog supporting output to Redis. If that's indeed the case you're going to need some kind of middle man.

Note that logstash-forwarder is a separate product that's only capable of reading text files and shipping them via the lumberjack protocol. There's no Redis support there.

1 Like
4

Thanks for your reply. So far I'm reading this: http://beanbag.technicalissues.us/2015/06/elk-stack-design/
Which discusses the issues I'm having.

5

Hello! I'm the original author of the omhiredis plugin for rsyslog, and I just stumbled onto this post. The original plugin was very simple and I mainly used it just for incrementing counters for stats tracking. However, I've made some improvements for the upcoming rsyslog 8.13 release, which should hopefully occur in sometime this month (September, 2015).

I have a blog post previewing the new functionality in detail including examples here: http://oldschool.systems/post/omhiredis/ .

I'd love to find some people who have a use case for the new features, as I would greatly enjoy bug reports and feedback!

Cheers,
Brian

6

@taotetek Thank you so much. I love the plugin.

Before

client ----> Rsyslog ---> logstash ----> redis ---> logstash ---> ES

now

client ----> Rsyslog ----> redis ---> logstash ---> ES

Much faster and also I can get my the IP of the server that ships the logs not 127.0.0.1 from the logstash shipper.

Regards,
Gabriel