Syslog-ng directly into Redis from which a central Logstash indexer can read

Elastic Stack Logstash
1

Hi,

Currently doing a feasability test of the ELK stack and Redis solution as a replacement for Splunk. I've done much googling and lot of testing but it's hard to run wild in a DevOps enviroment where everything is potentially used for production systems so I'm swalowing my pride and asking the internet for help :stuck_out_tongue:

For the feasability to be a sucess I have a requirement for a syslog-ng server with no forwarder to send data to a remote Logstash indexer but I'm concerned about the volume so want to use Redis as an intermediary buffer. My preferred solution would be to install a Logstash forwarder on the syslog-ng server so that the forwarder could send date to the Redis server for the centralised Logstash indexer to index at leisure but the syslog-ng server isn't permited to have anything else installed on it. I can however get the necessary configuration for the syslog-ng server to send messages directly to the centralised Logstash indexer or a remote Logstash forwarder added but that would not be good because of the volume and potentially dropped messages due to congestion (hence wanting to have syslog-ng communucate with Redis).

I've had confirmation that the version of syslog-ng running on the syslog-ng server supports sending messages directly to Redis. The general format according to google is:

destination d_redis {
    redis(
        host("redis-server.somedomain.org")
        port(6379)
        command("redis-command", "first-command-parameter", "second-command-parameter", "third-command-parameter")
    );
};
log {
    source(s_network);
    parser(p_network);
    destination(d_redis);
};

Ideally I would also like to keep the Logstash configuration pretty standard, like the below:

input {
  redis{
    host => "redis-server.somedomain.org"
    port => 6379
    data_type => "list"
    key => "logstash"
    codec => json
  }
}

Does anybody know the configuration I should request added as a distination on the syslog-ng configuration file to send messages deirectly to Redis in a reliable manner in a format compatible with the above Logstash input?

Thanks :smile:

2

Hi,

are you sure that you need Logstash in the pipeline? The latest version of syslog-ng can directly send messages to Redis and Elasticsearch (see Sending messages directly to Elasticsearch from syslog-ng).

Regards,

Robert Fekete
syslog-ng documentation maintainer

1 Like
3

Hi Robert,

Thank you for your reply.

I didn't know that syslog-ng could could send messages directly into elasticsearch. That will be usefull in the future.

Unfortunately for the time being Logstash is needed due to some preprocessing requirements and resource limitations of the syslog-ng server (which if we could do with the syslog-ng parser and syslog-ng into elasticsearch would be great but the issue is access to the syslog-ng server for maintence and not intereferring too much with live systems). Hence wanting to send syslog-ng into Redis (currently sending directly to Logstash indexer as TCP but I'm thinking about future load and removing additional complexity as far as possible [otherwise would of used a remote forwarder as an intermediary to Redis] so want to use direct into Redis like I do with forwarders).

:smiley:

4

Just wondering if you figured this out yet as I'd like to ship syslog-ng directly to redis also.

5

Hi genebean,

Not yet, curently concentrating on the stuff I can do but I will be looking more into it in a few weeks if no one provides a pre-baked solution here and will update this thread if successfull (I guess I will need to read a lot of examples on syslog-ng and redis before then). In the meantime I'm having syslog-ng send events to an instance of Logstash via TCP that is installed on the redis server with that instance of Logstash not doing any heavy processing (only adding a type field, time received and formatting into json) then inserting into redis for a more powerfull server with a centralised Logstash instance to later pull events from the redis server to heavily manipulate and index them into elasticsearch (and the redis server is very much needed as a buffer due to the volume). Obviously missing out that intemeiadiary Logstash instance would be better to reduce potentila issues in the future (I know syslog-ng can now send events directly into elasticsearch but Logstash is unfortunately required in our set up).

6

In your current setup, do the log entries have the original host when they get to ES or is it set to the syslog-ng server? If it's set correctly I'd appreciate it if you would share both the syslog-ng config and that of the Logstash instance inserting into Redis.

7

The original host is in elasticsearch using the logstash syslog filter. This is the basics of what to do with syslog messages once logstash receives them (courtesy of https://www.elastic.co/guide/en/logstash/current/config-examples.html):

input {
  tcp {
    port => 514
    type => syslog
  }
  udp {
    port => 514
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { host => localhost }
}

Syslog-ng into Logstash is a pretty standard thing, you can see an example here:

destination d_logstash {
        tcp("logstash.mydomain.org" port(514));
};

I don't want to go into too much detail in this forum post as lots of pre-existing posts/threads on this forum detail it better along with articles on the Internet as a pretty standard thing to do and so I want to try to keep this thread on topic (but hope the above has helped if you no longer want to send syslog-ng preformatted into redis which is what this thread is for). If no existing ones are enough please create a new topic and message me the link or post it here (to prevent this one getting highjacked) and I'll give you much more detail in there.

:smile:

8

Thanks for the info. Hopefully someone can help us get the direct to redis part working :smile:

9

After some reading tonight I think this destination setup will work in my syslog-ng.conf (still need to test though):

destination d_redis {
  host("redis.example.com")
  port("6379")
  command("rpush" "logstash" "{\"message\":\"<${PRI}>${ISODATE} ${HOST} ${LEVEL} ${MSGHDR}${MSG},\"logsource\":\"${HOST}\",\"host\":\"${LOGHOST}\",\"type\":\"syslog\",\"tags\":[\"forwarded_syslog\"]}")
};

The general format of this was determined by looking at the output of redis-cli monitor. The macros came from http://bit.ly/1MsfZM5. For me, I am trying to set logsource to the server that generated the original message and host to the syslog-ng server.

1 Like
10

Thanks. I'll give this a go next week when I can get it added to the syslog-ng server and feedback how it works out here, if you manage to do so before I can it would be great to know how you get on :smile:

11

FWIW Syslog's redis output does not support batch inserts. We found it to slow to use in production

12

Confirmed syslog-ng to Redis is too slow due to the volume of messages.

Used an intermediary Logstash forwarder in the end.

Thanks for all the input :slight_smile: