Hello all...
I am receiving logs from 5 different sources on one single port. In fact it is a collection of files being sent through syslog from a server in realtime. The server stores logs from 4 VPN servers and one DNS server. Now the server admin started sending all 5 types of files on a single port although I asked something different. Anyways, I thought to make this also work now.
Below are the different types of samples-
<134>Sep 30 22:02:27 mumvpn.domain1.com PulseSecure: id=firewall time="2021-09-30 22:02:27" pri=6 fw=xx2.21.140.100 vpn=ive user=user1 realm="google_auth" roles="" proto=auth src=1xx.87.92.246 dst= dstname= type=vpn op= arg="" result= sent= rcvd= agent="" duration= msg="AUT24803: Host Checker policy 'Hostchecker_Policy' passed on host '1xx.87.92.246' address '7a-fc-ac-08-88-9e' for user 'user1'."
------------------
<13>Sep 30 22:03:28 xx2.20.43.100 370 <134>1 2021-09-30T22:03:28+05:30 canopus.domain1.com1 PulseSecure: - - - id=firewall time="2021-09-30 22:03:28" pri=6 fw=xx2.20.43.100 vpn=ive user=System realm="google_auth" roles="" proto= src=1xx.99.110.19 dst= dstname= type=vpn op= arg="" result= sent= rcvd= agent="" duration= msg="AUT23278: User Limit realm restrictions successfully passed for /google_auth "
------------------
<134>Sep 30 22:41:43 xx2.20.43.101 1 2021-09-30T22:41:43+05:30 canopus.domain1.com2 PulseSecure: - - - id=firewall time="2021-09-30 22:41:43" pri=6 fw=xx2.20.43.101 vpn=ive user=user22 realm="google_auth" roles="Domain_check_role" proto= src=1xx.200.27.62 dst= dstname= type=vpn op= arg="" result= sent= rcvd= agent="" duration= msg="NWC24328: Transport mode switched over to SSL for user with NCIP xx2.20.210.252 "
------------------
<134>Sep 30 22:36:59 vpn-dns-1 named[130237]: 30-Sep-2021 22:36:59.172 queries: info: client @0x7f8e0f5cab50 xx2.30.16.147#63335 (ind.event.freefiremobile.com): query: ind.event.freefiremobile.com IN A + (xx2.31.0.171)
------------------
<13>Sep 30 22:40:31 xx2.20.43.101 394 <134>1 2021-09-30T22:40:31+05:30 canopus.domain1.com2 PulseSecure: - - - id=firewall time="2021-09-30 22:40:31" pri=6 fw=xx2.20.43.101 vpn=ive user=user3 realm="google_auth" roles="Domain_check_role" proto= src=1xx.168.77.166 dst= dstname= type=vpn op= arg="" result= sent= rcvd= agent="" duration= msg="NWC23508: Key Exchange number 1 occurred for user with NCIP xx2.20.214.109 "
Below is my config file-
input {
syslog {
port => 1301
ecs_compatibility => disabled
tags => ["vpn"]
}
}
I tried to apply a condition first to get VPN logs (1st sample logline) and pass it to dissect-
filter {
if "vpn" in [tags] {
#if ([message] =~ /vpn=ive/) {
if "vpn=ive" in [message] {
dissect {
mapping => { "message" => "%{reserved} id=firewall %{message1}" }
# using id=firewall to get KV pairs in message1
}
}
}
else { drop {} }
# \/ end of filter brace
}
But when I run with this config file, I am getting mixture of all 5 types of logs in kibana. I don't see any dissect failures as well.
Another question is, if I have to process all 5 types of logs in one config file, will below be a good approach?
if "VPN-logline" in [message] { use KV plugin and add tag of "vpn" }
else if "DNS-logline" in [message] { use JSON plugin and tag of "dns"}
else if "something-irrelevant" in [message] { drop {} }