Hello Explorers
Hope I could help you if you are looking to set up a Syslog-ng Logstash configuration to transfer logs from a Client server to Master server.
Advantage of this setup:
If you have 100 Client servers, and you need to check a Specific/Multiple log files across all the 100 servers everyday. It would be a tiresome job.
Using this setup you can push multiple logs from 100 client servers to a single Master server in to specific files. So logs of 100 client servers will be available in single Master server!!!
Architecture:
Here, we have Syslog-ng as "Shipper" in the client side and Logstash as the service in the Server side to receive the logs from Client server.
Pre-requisites:
- Here I have used 2 [CentOS-7-x86_64-Minimal-2003.iso] VM servers spun up using VMware.
Note: Mirror- (http://mirrors.piconets.webwerks.in/centos-mirror/7.8.2003/isos/x86_64/CentOS-7-x86_64-Minimal-2003.iso)
-
Configured Java-1.8.0-openjdk.x86_64, set Java home path on both machines.
-
Syslog-ng 3.29 on Client.
-
Logstash 7 in server side.
Lets have a look at this.
Scenario:1 To transfer logs from Syslog-ng to Logstash WITHOUT TLS over TCP.
Client-side Syslog-ng configuration
source s_file {
file("/var/log/messages");
};
destination d_syslog_tcp {
syslog("172.16.190.130" transport("tcp") port(6514));
};
log {
source(s_file);destination(d_syslog_tcp);
};
Server-side Logstash configuration
Note: Make sure that you have changed the the user and group names in the file /etc/systemd/system/logstash.service from "logstash" to "root"
input {
tcp {
port => 6514
mode => "server"
type => "syslog"
}
}
output {
if [type] == "syslog" {
file {
path => "/root/logs/%{host}/%{+YYYY-MM-dd}.log"
}
}
}
Scenario: 2 - To transfer logs from Syslog-ng to Logstash WITH TLS over TCP.
For creating the CA certificate, Client and Server certificate, please refer the official syslog-ng documentation:
CA and Server-side certificate creation =>
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/mutual-authentication-using-tls
Client side certificate creation: =>
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/mutual-authentication-using-tls/2#TOPIC-1430889
Configuring the certificates across the servers =>
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/mutual-authentication-using-tls/2#TOPIC-1430890
Client-side Syslog-ng TLS configuration
source s_file {
file("/var/log/messages");
};
destination d_tls_syslog {
syslog("172.16.190.130"
transport("tls")
port(6514)
tls(
key-file("/etc/syslog-ng/cert.d/clientkey.pem")
cert-file("/etc/syslog-ng/cert.d/clientreq.pem")
ca-dir("/etc/syslog-ng/ca.d")
peer-verify(no)
)
);
};
log {
source(s_file);destination(d_tls_syslog);
};
Server-side Logstash configuration
Note: Make sure that you have changed the the user and group names in the file /etc/systemd/system/logstash.service from "logstash" to "root"
input {
tcp {
port => 6514
mode => "server"
type => syslog
ssl_enable => true
ssl_certificate_authorities => "/etc/logstash/ca.d/cacert.pem"
ssl_cert => "/etc/logstash/serverreq.pem"
ssl_key => "/etc/logstash/serverkey.pem"
ssl_verify => false
}
}
output {
file {
path => "/root/tls_logs/%{host}/%{+YYYY-MM-dd}.log"
}
}