Grok filter for cyberoam syslogs

Elastic Stack Logstash
1

Hi all, i have been experementing with ELK the last 2 weeks. And it looks very good! But my main goal is to analyze our own firewall logs. But it dosnt look like my syslog filter can match any fields.

I didnt get any data with the syslog plugin so I guess the cyberoam syslog format isnt supported out of the box, So i'm using tcp/udp inputs and set the type to syslog manually.

input {
    tcp {
        port => 1514
        type => "syslog"
    }
    udp {
        port => 1514
        type => "syslog"
    }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
  }
}

output {
  if [type] == "syslog" and "_grokparsefailure" in [tags] {
    file { path => "failed_syslog_events-%{+YYYY-MM-dd}" }
  }
}

output {
    elasticsearch{
        host => localhost
    }
    stdout {}
}

using the http://grokconstructor.appspot.com/ i ended up with a filter looking like this

\A%{TIMESTAMP_ISO8601}%{SPACE}%{IPORHOST}%{SPACE}%{SYSLOG5424PRI}%{WORD}%{SYSLOG5424PRINTASCII}%{SPACE}%{SYSLOG5424PRINTASCII}%{SPACE}%{SYSLOG5424PRINTASCII}%{SPACE}%{SYSLOG5424PRINTASCII}%{SPACE}%{SYSLOG5424PRINTASCII}%{SPACE}%{SYSLOG5424PRINTASCII}%{SPACE}%{SYSLOG5424PRINTASCII}%{SPACE}

and that dosnt seem right at all :slight_smile:

logdata i try to get into fields:
:arrow_forward: tail /var/log/logstash/logstash.stdout

2015-07-15T12:06:15.292Z 10.172.200.254 <30>date=2015-07-15 time=14:06:15 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=010302602002 log_type="Firewall" log_component="Appliance Access" log_subtype="Denied" status="Deny" priority=Information duration=0 fw_rule_id=0 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" in_interface="PortA.301" out_interface="" src_mac=c4:57:6e:33:67:5e src_ip=10.19.133.133 src_country_code= dst_ip=10.19.133.255 dst_country_code= protocol="UDP" src_port=46795 dst_port=58581 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip= tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir_disp="" connid="" vconnid=""
2015-07-15T12:06:15.524Z 10.172.200.254 <30>date=2015-07-15 time=14:06:15 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=010302602002 log_type="Firewall" log_component="Appliance Access" log_subtype="Denied" status="Deny" priority=Information duration=0 fw_rule_id=0 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" in_interface="PortA.301" out_interface="" src_mac=44:39:c4:53:e4:b5 src_ip=10.19.133.190 src_country_code= dst_ip=10.19.133.255 dst_country_code= protocol="UDP" src_port=57621 dst_port=57621 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip= tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir_disp="" connid="" vconnid=""
2015-07-15T12:06:15.573Z 10.172.200.254 <30>date=2015-07-15 time=14:06:15 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=050901616001 log_type="Content Filtering" log_component="HTTP" log_subtype="Allowed" status="" priority=Information fw_rule_id=0 user_name="" user_gp="" iap=5 category="None" category_type="" url="*.systemmonitor.eu.com" contenttype="" httpresponsecode="" src_ip=172.172.150.10 dst_ip=31.222.134.218 protocol="TCP" src_port=61163 dst_port=443 sent_bytes=0 recv_bytes=8152 domain=*.systemmonitor.eu.com
2015-07-15T12:06:16.216Z 10.172.200.254 <30>date=2015-07-15 time=14:06:16 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=050901616001 log_type="Content Filtering" log_component="HTTP" log_subtype="Allowed" status="" priority=Information fw_rule_id=0 user_name="" user_gp="" iap=5 category="None" category_type="" url="*.systemmonitor.eu.com" contenttype="" httpresponsecode="" src_ip=172.172.150.10 dst_ip=31.222.134.219 protocol="TCP" src_port=61165 dst_port=443 sent_bytes=0 recv_bytes=7016 domain=*.systemmonitor.eu.com
2015-07-15T12:06:17.264Z 10.172.200.254 <30>date=2015-07-15 time=14:06:17 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=010302602002 log_type="Firewall" log_component="Appliance Access" log_subtype="Denied" status="Deny" priority=Information duration=0 fw_rule_id=0 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" in_interface="PortD" out_interface="" src_mac=00:1e:7a:e5:d6:98 src_ip=172.22.117.162 src_country_code= dst_ip=172.31.143.66 dst_country_code= protocol="TCP" src_port=64684 dst_port=8124 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip= tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir_disp="" connid="" vconnid=""
2015-07-15T12:06:17.264Z 10.172.200.254 <30>date=2015-07-15 time=14:06:17 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=010302602002 log_type="Firewall" log_component="Appliance Access" log_subtype="Denied" status="Deny" priority=Information duration=0 fw_rule_id=0 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" in_interface="PortA.301" out_interface="" src_mac=c4:57:6e:33:67:5e src_ip=10.19.133.133 src_country_code= dst_ip=10.19.133.255 dst_country_code= protocol="UDP" src_port=46795 dst_port=58581 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip= tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir_disp="" connid="" vconnid=""
2015-07-15T12:06:19.292Z 10.172.200.254 <30>date=2015-07-15 time=14:06:19 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=010302602002 log_type="Firewall" log_component="Appliance Access" log_subtype="Denied" status="Deny" priority=Information duration=0 fw_rule_id=0 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" in_interface="PortA.301" out_interface="" src_mac=c4:57:6e:33:67:5e src_ip=10.19.133.133 src_country_code= dst_ip=10.19.133.255 dst_country_code= protocol="UDP" src_port=46795 dst_port=58581 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip= tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir_disp="" connid="" vconnid=""
2015-07-15T12:06:21.292Z 10.172.200.254 <30>date=2015-07-15 time=14:06:21 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=010302602002 log_type="Firewall" log_component="Appliance Access" log_subtype="Denied" status="Deny" priority=Information duration=0 fw_rule_id=0 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" in_interface="PortA.301" out_interface="" src_mac=c4:57:6e:33:67:5e src_ip=10.19.133.133 src_country_code= dst_ip=10.19.133.255 dst_country_code= protocol="UDP" src_port=46795 dst_port=58581 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip= tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir_disp="" connid="" vconnid=""
2015-07-15T12:06:23.267Z 10.172.200.254 <30>date=2015-07-15 time=14:06:23 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=010302602002 log_type="Firewall" log_component="Appliance Access" log_subtype="Denied" status="Deny" priority=Information duration=0 fw_rule_id=0 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" in_interface="PortD" out_interface="" src_mac=00:1e:7a:e5:d6:98 src_ip=172.22.117.162 src_country_code= dst_ip=172.31.143.66 dst_country_code= protocol="TCP" src_port=64684 dst_port=8124 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip= tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir_disp="" connid="" vconnid=""
2015-07-15T12:06:23.268Z 10.172.200.254 <30>date=2015-07-15 time=14:06:23 timezone="CEST" device_name="CR25iNG" device_id=C06114310926-94D8P8 log_id=010302602002 log_type="Firewall" log_component="Appliance Access" log_subtype="Denied" status="Deny" priority=Information duration=0 fw_rule_id=0 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" in_interface="PortA.301" out_interface="" src_mac=c4:57:6e:33:67:5e src_ip=10.19.133.133 src_country_code= dst_ip=10.19.133.255 dst_country_code= protocol="UDP" src_port=46795 dst_port=58581 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip= tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir_disp="" connid="" vconnid=""

Can anybody help me get this kind of data into fields i can use in Kibana?

2

after i discovered the kv plugin it was dead easy to parse the log:

input {
  tcp {
    port => 1514
    type => "syslog"
  }
  udp {
    port => 1514
    type => "syslog"
  }
} 

filter {
  if [type] == "syslog" {
    kv {
      source => "message"
    }
  }
}
  
output {
  elasticsearch{
    host => localhost
  }
  stdout {}
}
1 Like
3

I am also in Testing phase with my ELK cluster, I have tested and used grok filters for various other devices such as Windows, Linux, Cisco switches,routers and ASA firewall. Now I am in attempt to write a generate a grok for Cyberoam Device as well.

I landed here searching for the same, As this thread is pretty much old, I wanted to know if you have solved the issue / generated a filter for cyberoam that I can use for my tests.