Parsing my Zyxel USG110 CEF syslog file

dlex51 · August 29, 2017, 2:48pm

Hello everybody,

i'm new to ELK Stack, but with some exceptions everything worked (installation) like a charme.

Now I'm trying to visualize my router logs with Kibana. Therefore i receive the syslogs from my router via rsyslogd in the cef/syslog format.

Unfortunately I'm completly new to the filtering things. Can you help me to create a filter for logs like:

Aug 29 15:39:47 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|Access Control|3|src=xx.xx.xx.xx dst=xx.xx.xx.xx spt=443 dpt=62872 msg=invalid state detected, DROP proto=6 app=others
Aug 29 15:39:47 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|Access Control|5|src=xx.xx.xx.xx dst=xx.xx.xx.xx spt=138 dpt=138 msg=Match default rule, DROP proto=17 app=others
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: got_one: receive packet from raw socket.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: got_one: send packet to do_packet function.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: do_packet: Receive packet.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: do_packet: Prepare to parse option buffer.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: do_packet: dhcp function called.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: do_packet: This packet is from 0.0.0.0 via 60:31:97:7e:31:cb.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: dhcp: Receive DISCOVER.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: [dhcpdiscover]Enforce find_lease().
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: Found host for link address: 192.168.2.100.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: find_lease: Requested address not available:erro2.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: choosing fixed address.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: Returning lease: 192.168.2.100.
Aug 29 15:39:48 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: DHCPDISCOVER from 60:31:97:7e:31:cb via eth4
Aug 29 15:39:49 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: DHCPOFFER on 192.168.2.100 to 60:31:97:7e:31:cb via eth4
Aug 29 15:39:49 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|Access Control|3|src=10.0.30.38 dst=xx.xx.xx.xx spt=41575 dpt=443 msg=invalid state detected, DROP proto=6 app=others
Aug 29 15:39:50 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|Access Control|3|src=192.168.2.64 dst=xx.xx.xx.xx spt=52421 dpt=443 msg=invalid state detected, DROP proto=6 app=others
Aug 29 15:39:51 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|Access Control|3|src=192.168.2.86 dst=xx.xx.xx.xx spt=56512 dpt=443 msg=invalid state detected, DROP proto=6 app=others
Aug 29 15:39:51 usg110 CEF: message repeated 2 times: [0|ZyXEL|USG110|4.25(AAPH.0)|0|Access Control|3|src=192.168.2.86 dst=xx.xx.xx.xx spt=56512 dpt=443 msg=invalid state detected, DROP proto=6 app=others]
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|Access Control|3|src=10.0.30.86 dst=216.58.206.4 spt=45936 dpt=443 msg=invalid state detected, DROP proto=6 app=others
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|Access Control|3|src=192.168.2.86 dst=xx.xx.xx.xx spt=56512 dpt=443 msg=invalid state detected, DROP proto=6 app=others
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: got_one: receive packet from raw socket.
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: got_one: send packet to do_packet function.
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: do_packet: This packet is from 0.0.0.0 via 60:31:97:7e:31:cb.
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: dhcp: Receive DISCOVER.
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: [dhcpdiscover]Enforce find_lease().
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: Found host for link address: 192.168.2.100.
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: find_lease: Requested address not available:erro2.
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: choosing fixed address.
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: Returning lease: 192.168.2.100.
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: DHCPDISCOVER from 60:31:97:7e:31:cb via eth4
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: ack_lease: Need to send ICMP to check this OFFER.
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: ack_lease: Ping timeout: 1
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110||0|Default|3|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=dhcpd: unexpected ICMP Echo Reply from 192.168.2.100
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|INTERFACE STATISTICS|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=name=Port1,status=1000M/Full,TxPkts=219668946,RxPkts=2581639353,Colli.=0,TxB/s=123055,RxB/s=745707,UpTime=699:50:22
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|INTERFACE STATISTICS|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=name=Port2,status=1000M/Full,TxPkts=0,RxPkts=1635918,Colli.=0,TxB/s=0,RxB/s=0,UpTime=2710:15:25
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|INTERFACE STATISTICS|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=name=Port3,status=Down,TxPkts=0,RxPkts=0,Colli.=0,TxB/s=0,RxB/s=0,UpTime=00:00:00
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|INTERFACE STATISTICS|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=name=Port4,status=1000M/Full,TxPkts=1580880617,RxPkts=247242851,Colli.=0,TxB/s=515224,RxB/s=5850993,UpTime=2710:15:24
Aug 29 15:39:52 usg110 CEF: 0|ZyXEL|USG110|4.25(AAPH.0)|0|INTERFACE STATISTICS|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=name=Port5,status=1000M/Full,TxPkts=2979698627,RxPkts=1834377450,Colli.=0,TxB/s=6892502,RxB/s=191809,UpTime=2710:15:24

Thanks in advance!

Best Regards,
dlex51

dlex51 · August 31, 2017, 10:05am

Hello everybody,

acutally i can also switch the routers config not to send syslog in CEF format, so the logs would look like this:

Aug 31 11:02:38 2017 usg110 src="0.0.0.0:0" dst="0.0.0.0:0" msg="dhcpd: find_lease: Requested address not available:erro2." note="" user="unknown" devID="a0e4cb7ce9d7" cat="Default"
Aug 31 11:02:38 2017 usg110 src="0.0.0.0:0" dst="0.0.0.0:0" msg="dhcpd: choosing fixed address." note="" user="unknown" devID="a0e4cb7ce9d7" cat="Default"
Aug 31 11:02:38 2017 usg110 src="0.0.0.0:0" dst="0.0.0.0:0" msg="dhcpd: Returning lease: 192.168.2.100." note="" user="unknown" devID="a0e4cb7ce9d7" cat="Default"
Aug 31 11:02:38 2017 usg110 src="0.0.0.0:0" dst="0.0.0.0:0" msg="dhcpd: DHCPDISCOVER from 60:31:97:7e:31:cb via eth4" note="" user="unknown" devID="a0e4cb7ce9d7" cat="Default"
Aug 31 11:02:38 2017 usg110 src="0.0.0.0:0" dst="0.0.0.0:0" msg="dhcpd: ack_lease: Need to send ICMP to check this OFFER." note="" user="unknown" devID="a0e4cb7ce9d7" cat="Default"
Aug 31 11:02:38 2017 usg110 src="0.0.0.0:0" dst="0.0.0.0:0" msg="dhcpd: ack_lease: Ping timeout: 1" note="" user="unknown" devID="a0e4cb7ce9d7" cat="Default"
Aug 31 11:02:38 2017 usg110 src="0.0.0.0:0" dst="0.0.0.0:0" msg="dhcpd: unexpected ICMP Echo Reply from 192.168.2.100" note="" user="unknown" devID="a0e4cb7ce9d7" cat="Default"
Aug 31 11:02:38 2017 usg110 src="192.168.2.129:53087" dst="xx.xx.xx.xx:443" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="d.schneider" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others"
Aug 31 11:02:39 2017 usg110 src="0.0.0.0:0" dst="0.0.0.0:0" msg="DHCP server offered 192.168.2.100 to (60:31:97:7E:31:CB)" note="DHCP Offer" user="unknown" devID="a0e4cb7ce9d7" cat="DHCP"
Aug 31 11:02:39 2017 usg110 src="0.0.0.0:0" dst="0.0.0.0:0" msg="dhcpd: DHCPOFFER on 192.168.2.100 to 60:31:97:7e:31:cb via eth4" note="" user="unknown" devID="a0e4cb7ce9d7" cat="Default"
Aug 31 11:02:39 2017 usg110 src="104.84.170.180:443" dst="xx.xx.xx.xx:50134" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others"
Aug 31 11:02:39 2017 usg110 src="104.84.170.180:443" dst="xx.xx.xx.xx:50134" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others"
Aug 31 11:02:40 2017 usg110 src="104.84.170.180:443" dst="xx.xx.xx.xx:50143" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others"
Aug 31 11:02:40 2017 usg110 src="104.84.170.180:443" dst="xx.xx.xx.xx:50143" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others"
Aug 31 11:02:40 2017 usg110 src="192.168.2.83:63794" dst="162.125.66.3:443" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others"
Aug 31 11:02:40 2017 usg110 src="10.0.30.58:0" dst="10.0.30.1:0" msg="Match default rule, ICMP Type:8, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=1 proto="others"
Aug 31 11:02:40 2017 usg110 src="104.84.170.180:443" dst="xx.xx.xx.xx:50149" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others"
Aug 31 11:02:40 2017 usg110 src="104.84.170.180:443" dst="xx.xx.xx.xx:50149" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others"
Aug 31 11:02:40 2017 usg110 src="192.168.0.1:0" dst="224.0.0.1:0" msg="Match default rule, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=2 proto="others"
Aug 31 11:02:41 2017 usg110 src="104.84.170.180:443" dst="xx.xx.xx.xx:50154" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="a0e4cb7ce9d7" cat="Security Policy Control" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others"

Maybe somebody could tell me how to parse, process and visualize these kind of logs?

Best Regards,
dlex51

guyboertje · September 4, 2017, 3:22pm

Logstash has a CEF codec.

What input are you using?

dlex51 · September 5, 2017, 6:03am

Hey guybortje,

I ship the router logs against rsyslog and afterwards i use file input to get it into logstash.

guyboertje · September 5, 2017, 1:32pm

You will not be able to use the CEF codec with the file input because you will need a cef_lines codec that does not exist.

The CEF codec does a fancy de-structuring exercise on the line of data which caters for several variants. You do not have to deal with variants.

Use the plain codec on the file input and do the de-structuring with grok or dissect filters.

dlex51 · September 5, 2017, 1:51pm

Ok, i see. Hmm can you send me an example how to do it?

guyboertje · September 5, 2017, 2:20pm

Here is a primer
https://rolinh.ch/posts/2017/03/12/quick-logstash-grok-patterns-testing/

dlex51 · September 6, 2017, 1:51pm

Thanks a lot mate. I will try to parse it with those patterns.

system · October 4, 2017, 1:52pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.