# Syslog-ng directly into Redis from which a central Logstash indexer can read
**URL:** https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590
**Category:** Logstash
**Created:** [October 20, 2015, 4:11pm UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590 "2015-10-20T16:11:59Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@ottignon](https://discuss.elastic.co/u/ottignon)
#### Post date: [October 20, 2015, 4:11pm UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/1 "2015-10-20T16:11:59Z")
Hi,
Currently doing a feasability test of the ELK stack and Redis solution as a replacement for Splunk. I've done much googling and lot of testing but it's hard to run wild in a DevOps enviroment where everything is potentially used for production systems so I'm swalowing my pride and asking the internet for help π
For the feasability to be a sucess I have a requirement for a syslog-ng server with no forwarder to send data to a remote Logstash indexer but I'm concerned about the volume so want to use Redis as an intermediary buffer. My preferred solution would be to install a Logstash forwarder on the syslog-ng server so that the forwarder could send date to the Redis server for the centralised Logstash indexer to index at leisure but the syslog-ng server isn't permited to have anything else installed on it. I can however get the necessary configuration for the syslog-ng server to send messages directly to the centralised Logstash indexer or a remote Logstash forwarder added but that would not be good because of the volume and potentially dropped messages due to congestion (hence wanting to have syslog-ng communucate with Redis).
I've had confirmation that the version of syslog-ng running on the syslog-ng server supports sending messages directly to Redis. The general format according to google is:
```
destination d_redis {
redis(
host("redis-server.somedomain.org")
port(6379)
command("redis-command", "first-command-parameter", "second-command-parameter", "third-command-parameter")
);
};
log {
source(s_network);
parser(p_network);
destination(d_redis);
};
```
Ideally I would also like to keep the Logstash configuration pretty standard, like the below:
```
input {
redis{
host => "redis-server.somedomain.org"
port => 6379
data_type => "list"
key => "logstash"
codec => json
}
}
```
Does anybody know the configuration I should request added as a distination on the syslog-ng configuration file to send messages deirectly to Redis in a reliable manner in a format compatible with the above Logstash input?
Thanks π
---
### Author:  [@fekete\_robert](https://discuss.elastic.co/u/fekete_robert)
#### Post date: [October 22, 2015, 6:09am UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/2 "2015-10-22T06:09:48Z")
Hi,
are you sure that you need Logstash in the pipeline? The latest version of syslog-ng can directly send messages to Redis and Elasticsearch (see [Sending messages directly to Elasticsearch from syslog-ng](https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-elasticsearch.html)).
Regards,
Robert Fekete
syslog-ng documentation maintainer
---
### Author:  [@ottignon](https://discuss.elastic.co/u/ottignon)
#### Post date: [October 22, 2015, 12:25pm UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/3 "2015-10-22T12:25:54Z")
Hi Robert,
Thank you for your reply.
I didn't know that syslog-ng could could send messages directly into elasticsearch. That will be usefull in the future.
Unfortunately for the time being Logstash is needed due to some preprocessing requirements and resource limitations of the syslog-ng server (which if we could do with the syslog-ng parser and syslog-ng into elasticsearch would be great but the issue is access to the syslog-ng server for maintence and not intereferring too much with live systems). Hence wanting to send syslog-ng into Redis (currently sending directly to Logstash indexer as TCP but I'm thinking about future load and removing additional complexity as far as possible [otherwise would of used a remote forwarder as an intermediary to Redis] so want to use direct into Redis like I do with forwarders).
π
---
### Author:  [@genebean](https://discuss.elastic.co/u/genebean)
#### Post date: [October 30, 2015, 3:32am UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/4 "2015-10-30T03:32:57Z")
Just wondering if you figured this out yet as I'd like to ship syslog-ng directly to redis also.
---
### Author:  [@ottignon](https://discuss.elastic.co/u/ottignon)
#### Post date: [October 30, 2015, 11:12am UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/5 "2015-10-30T11:12:55Z")
Hi genebean,
Not yet, curently concentrating on the stuff I can do but I will be looking more into it in a few weeks if no one provides a pre-baked solution here and will update this thread if successfull (I guess I will need to read a lot of examples on syslog-ng and redis before then). In the meantime I'm having syslog-ng send events to an instance of Logstash via TCP that is installed on the redis server with that instance of Logstash not doing any heavy processing (only adding a type field, time received and formatting into json) then inserting into redis for a more powerfull server with a centralised Logstash instance to later pull events from the redis server to heavily manipulate and index them into elasticsearch (and the redis server is very much needed as a buffer due to the volume). Obviously missing out that intemeiadiary Logstash instance would be better to reduce potentila issues in the future (I know syslog-ng can now send events directly into elasticsearch but Logstash is unfortunately required in our set up).
---
### Author:  [@genebean](https://discuss.elastic.co/u/genebean)
#### Post date: [October 30, 2015, 11:34am UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/6 "2015-10-30T11:34:07Z")
In your current setup, do the log entries have the original host when they get to ES or is it set to the syslog-ng server? If it's set correctly I'd appreciate it if you would share both the syslog-ng config and that of the Logstash instance inserting into Redis.
---
### Author:  [@ottignon](https://discuss.elastic.co/u/ottignon)
#### Post date: [October 30, 2015, 11:53am UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/7 "2015-10-30T11:53:58Z")
The original host is in elasticsearch using the logstash syslog filter. This is the basics of what to do with syslog messages once logstash receives them (courtesy of [https://www.elastic.co/guide/en/logstash/current/config-examples.html](https://www.elastic.co/guide/en/logstash/current/config-examples.html)):
```
input {
tcp {
port => 514
type => syslog
}
udp {
port => 514
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => ["received_at", "%{@timestamp}"]
add_field => ["received_from", "%{host}"]
}
syslog_pri { }
date {
match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
}
}
}
output {
elasticsearch { host => localhost }
}
```
Syslog-ng into Logstash is a pretty standard thing, you can see an example here:
> **[Playing with Logstash and syslog-ng: forwarding syslog traffic from a log server](http://fishnix.tumblr.com/post/31199329713/playing-with-logstash-and-syslog-ng-forwarding#.VjNeZit4mNo)**
>
> I, like most of you (hopefully) have a central syslog server. Itβs truly invaluable whether you have 3 servers or 300 servers. The combination of technical curiosity and the firehose of messages from...
```
destination d_logstash {
tcp("logstash.mydomain.org" port(514));
};
```
I don't want to go into too much detail in this forum post as lots of pre-existing posts/threads on this forum detail it better along with articles on the Internet as a pretty standard thing to do and so I want to try to keep this thread on topic (but hope the above has helped if you no longer want to send syslog-ng preformatted into redis which is what this thread is for). If no existing ones are enough please create a new topic and message me the link or post it here (to prevent this one getting highjacked) and I'll give you much more detail in there.
π
---
### Author:  [@genebean](https://discuss.elastic.co/u/genebean)
#### Post date: [November 2, 2015, 4:30am UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/8 "2015-11-02T04:30:19Z")
Thanks for the info. Hopefully someone can help us get the direct to redis part working π
---
### Author:  [@genebean](https://discuss.elastic.co/u/genebean)
#### Post date: [November 2, 2015, 5:08am UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/9 "2015-11-02T05:08:33Z")
After some reading tonight I think this destination setup will work in my syslog-ng.conf (still need to test though):
```
destination d_redis {
host("redis.example.com")
port("6379")
command("rpush" "logstash" "{\"message\":\"<${PRI}>${ISODATE} ${HOST} ${LEVEL} ${MSGHDR}${MSG},\"logsource\":\"${HOST}\",\"host\":\"${LOGHOST}\",\"type\":\"syslog\",\"tags\":[\"forwarded_syslog\"]}")
};
```
The general format of this was determined by looking at the output of `redis-cli monitor`. The macros came from [http://bit.ly/1MsfZM5](http://bit.ly/1MsfZM5). For me, I am trying to set `logsource` to the server that generated the original message and `host` to the syslog-ng server.
---
### Author:  [@ottignon](https://discuss.elastic.co/u/ottignon)
#### Post date: [November 2, 2015, 7:29pm UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/10 "2015-11-02T19:29:49Z")
Thanks. I'll give this a go next week when I can get it added to the syslog-ng server and feedback how it works out here, if you manage to do so before I can it would be great to know how you get on π
---
### Author:  [@Johntdyer](https://discuss.elastic.co/u/Johntdyer)
#### Post date: [December 28, 2015, 5:33pm UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/11 "2015-12-28T17:33:13Z")
FWIW Syslog's redis output does not support batch inserts. We found it to slow to use in production
---
### Author:  [@ottignon](https://discuss.elastic.co/u/ottignon)
#### Post date: [January 13, 2016, 4:18pm UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/12 "2016-01-13T16:18:44Z")
Confirmed syslog-ng to Redis is too slow due to the volume of messages.
Used an intermediary Logstash forwarder in the end.
Thanks for all the input π
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:15am UTC](https://discuss.elastic.co/t/syslog-ng-directly-into-redis-from-which-a-central-logstash-indexer-can-read/32590/13 "2017-07-06T05:15:52Z")