Dealing with varied log format

Hello!

I just finished reading the Logstash Book (great introduction to the subject!) and would like to start experimenting with proper log collection in my environment. I would like to start off with out firewall. Getting the stack together to collect logs has not been an issue, it was smooth sailing. I'm however not certain how I should deal with the logs that I'm collecting.
Below I have pasted some examples of how the logs might look, these are all from the same device. I've written a basic pattern which separate all the common denominators from the rest of the message. In the end my goal will be to visualise what rules are being hit the most, where connections are coming from (geoip) and generally just having tidy logs to search through when the need arises. How would I best go about dealing with this?

Log examples

<133>[2017-03-03 16:06:02] EFW: TCP_FLAG: prio=2 id=03300004 rev=1 event=tcp_flag_set action=strip_flag bad_flag=ECN rule=TCPECN recvif=interface srcip=10.1.1.1 destip=10.3.0.1 ipdf=1 ipproto=TCP ipdatalen=32 srcport=13111 destport=80 tcphdrlen=32 syn=1 ece=1 cwr=1
<134>[2017-03-03 16:06:02] EFW: CONN: prio=1 id=00600004 rev=1 event=conn_open_natsat rule=nat conn=open connipproto=TCP connrecvif=interface connsrcip=91.23.63.13 connsrcport=13111 conndestif=interface conndestip=50.19.91.84 conndestport=80 connnewsrcip=7.34.1.91 connnewsrcport=19057 connnewdestip=50.10.20.10 connnewdestport=80
<132>[2017-03-03 16:07:07] EFW: RULE: prio=3 id=06000051 rev=1 event=ruleset_drop_packet action=drop rule=Default_Rule recvif=interface srcip=123.123.123.23 destip=5.57.246.124 ipproto=TCP ipdatalen=20 srcport=58794 destport=23 syn=1
<132>[2017-03-03 16:07:07] EFW: RULE: prio=3 id=06000051 rev=1 event=ruleset_drop_packet action=drop rule=Default_Rule recvif=interface srcip=10.2.2.2 destip=20.1.2.3 ipproto=UDP ipdatalen=56 srcport=55258 destport=161 udptotlen=56
<132>[2017-03-03 16:07:06] EFW: RULE: prio=3 id=06000051 rev=1 event=ruleset_drop_packet action=drop rule=Default_Rule recvif=interface srcip=11.2.3.4 destip=3.6.1.3 ipproto=ICMP ipdatalen=8 icmptype=ECHO_REQUEST echoid=2 echoseq=46304
<132>[2017-03-03 16:08:10] EFW: ARP: prio=3 id=00300049 rev=1 event=invalid_arp_sender_ip_address action=drop rule=Default_Access_Rule recvif=interface hwsender=ff-ff-ff-ff-ff-ff hwdest=ff-ff-ff-ff-ff-ff arp=request srcenet=ff-ff-ff-ff-ff-ff srcip=10.2.1.4 destenet=00-00-00-00-00-00 destip=20.1.3.1
<134>[2017-03-03 16:08:16] EFW: CONN: prio=1 id=00600004 rev=1 event=conn_open_natsat rule=nat conn=open connipproto=UDP connrecvif=interface connsrcip=18.23.1.4 connsrcport=65122 conndestif=interface conndestip=1.5.1.7 conndestport=161 connnewsrcip=4.4.4.4 connnewsrcport=58367 connnewdestip=7.3.1.8 connnewdestport=161

Pattern

CLAVISTER \[%{DATA:timestamp}\] %{WORD:efw}: %{WORD:logtype}: prio=%{POSINT:prio} id=%{INT:id} rev=%{POSINT:rev} event=%{WORD:event} %{GREEDYDATA:message}

Thanks!

Have a look at the kv filter plugin.
This might be the one-minute introduction you look for: http://stackoverflow.com/a/31884354

1 Like

That worked a treat! Thanks a lot. :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.