Sending logs from Syslog-ng to Logstash with and without TLS

[Vishnuprasad]

Hello Explorers

Hope I could help you if you are looking to set up a Syslog-ng Logstash configuration to transfer logs from a Client server to Master server.

Advantage of this setup:

If you have 100 Client servers, and you need to check a Specific/Multiple log files across all the 100 servers everyday. It would be a tiresome job.

Using this setup you can push multiple logs from 100 client servers to a single Master server in to specific files. So logs of 100 client servers will be available in single Master server!!!

Architecture:

Here, we have Syslog-ng as "Shipper" in the client side and Logstash as the service in the Server side to receive the logs from Client server.

Pre-requisites:

1. Here I have used 2 [CentOS-7-x86_64-Minimal-2003.iso] VM servers spun up using VMware.

Note: Mirror- (http://mirrors.piconets.webwerks.in/centos-mirror/7.8.2003/isos/x86_64/CentOS-7-x86_64-Minimal-2003.iso)

2. Configured Java-1.8.0-openjdk.x86_64, set Java home path on both machines.

3. Syslog-ng 3.29 on Client.

4. Logstash 7 in server side.

Lets have a look at this.

Scenario:1 To transfer logs from Syslog-ng to Logstash WITHOUT TLS over TCP.

Client-side Syslog-ng configuration

source s_file {
    file("/var/log/messages");
};


destination d_syslog_tcp {
syslog("172.16.190.130" transport("tcp") port(6514));
 };


log {
    source(s_file);destination(d_syslog_tcp);
};

Server-side Logstash configuration
Note: Make sure that you have changed the the user and group names in the file /etc/systemd/system/logstash.service from "logstash" to "root"

input {
 tcp {
    port => 6514
    mode => "server"
    type => "syslog"
  }
}

output {
   if [type] == "syslog" {

      file {
          path => "/root/logs/%{host}/%{+YYYY-MM-dd}.log"
      }
   }
}

Scenario: 2 - To transfer logs from Syslog-ng to Logstash WITH TLS over TCP.

For creating the CA certificate, Client and Server certificate, please refer the official syslog-ng documentation:

CA and Server-side certificate creation =>
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/mutual-authentication-using-tls

Client side certificate creation: =>
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/mutual-authentication-using-tls/2#TOPIC-1430889

Configuring the certificates across the servers =>
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/mutual-authentication-using-tls/2#TOPIC-1430890

Client-side Syslog-ng TLS configuration

source s_file {
    file("/var/log/messages");
};


destination d_tls_syslog {
syslog("172.16.190.130"
  transport("tls")
  port(6514)
  tls(
    key-file("/etc/syslog-ng/cert.d/clientkey.pem")
    cert-file("/etc/syslog-ng/cert.d/clientreq.pem")
    ca-dir("/etc/syslog-ng/ca.d")
    peer-verify(no)
    )
  );

};



log {
    source(s_file);destination(d_tls_syslog);
};

Server-side Logstash configuration
Note: Make sure that you have changed the the user and group names in the file /etc/systemd/system/logstash.service from "logstash" to "root"

input {
 tcp {
  port => 6514
  mode => "server"
  type => syslog
  ssl_enable => true
  ssl_certificate_authorities => "/etc/logstash/ca.d/cacert.pem"
  ssl_cert => "/etc/logstash/serverreq.pem"
  ssl_key => "/etc/logstash/serverkey.pem"
  ssl_verify => false
 }
}

output {

      file {
          path => "/root/tls_logs/%{host}/%{+YYYY-MM-dd}.log"
      }
   }