Log Parsing

Elastic Stack Logstash
1

I am collecting hundred types of router logs from a syslog server . I was able to write grok pattern for few of them and feed them into elasticsearch and kibana. But its becoming a tedious job to write for each type of log. Can anyone please tell me a better way to parse logs in logstash?
Below are few types of logs:- (There are many other types of logs)

Ex:- 200.65.181.67 RP/0/RSP0/CPU0:Dec 3 10:51:15 IST: ntpd[266]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 15->2.

Ex:-212.65.181.67 LC/0/2/CPU0:Dec 3 11:07:38 IST: bfd_agent[128]: %L2-BFD-6-SESSION_STATE_UP : BFD session to neighbor 10.141.0.2 on interface TenGigE0/2/0/0.141142 is up.

2

What do your patterns look like for these?

4

1.100.65.181.67 RP/0/RSP0/CPU0:Nov 22 17:58:59 IST: SSHD_[65794]: %SECURITY-SSHD-6-INFO_SUCCESS_COMMAND : Successfully authenticated user 'nieg' from '100.65.181.42' on 'vty0'(cipher 'aes128-ctr', mac 'hmac-sha1', command 'show mpls forwarding prefix 6.6.6.6/32')

GROK PATTERN:%{IPV4:ROUTERIP}\s%{WIN3}[:]%{MONTH}\s%{MONTHDAY}\s%{TIME}\s%{TZ1}[:]\s%{PROCESSNAME}\S[%{POSINT:PROCESSID}][:]\s[%]%{WORD:MESSAGECATEGORY}%{WORD-GROUP}%{POSINT:SEVERITYLEVEL}%{DATA:MESSAGECODE}\s[:]%{DATA:MESSAGE}[\s][']%{WORD:USER}[']\s.*\s[']%{IPV4:SOURCEIP}[']

2.100.65.181.67 LC/0/0/CPU0:Nov 23 17:40:15 IST: arp[120]: %IP-ARP-6-INFO_DUPADDR : 'Duplicate IP address 10.0.10.13 on TenGigE0/0/0/1.2083, sourced by 000c.2920.1d8c'

GROK PATTERN:%{IPV4:ROUTERIP}\s%{WIN3}[:]%{MONTH}\s%{MONTHDAY}\s%{TIME}\s%{TZ1}[:][\s]%{PROCESSNAME}[%{POSINT:PROCESSID}][:]\s[%]%{WORD:MESSAGECATEGORY}%{WORD:GROUPID}%{POSINT:SEVERITYLEVEL}%{DATA:MESSAGECODE}\s[:]\s([']%{GREEDYDATA:MESSAGE}\s%{IPV4:DUPLICATEIP}\s%{NOTSPACE}\s%{DATA:INTERFACE}?%{NOTSPACE}\s%{MAC:SOURCEMAC}['])?

  1. 100.65.181.67 RP/0/RSP0/CPU0:Nov 28 12:33:01 IST: SSHD_[65793]: %SECURITY-SSHD-4-WARNING_LOGIN : The requested term-type 'dumb' is not supported

GROK PATTERN:%{IPV4:ROUTERIP}\s%{WIN3}[:]%{MONTH}\s%{MONTHDAY}\s%{TIME}\s%{TZ1}[:]\s%{PROCESSNAME}\S[%{POSINT:PROCESSID}][:]\s[%]%{WORD:MESSAGECATEGORY}%{WORD:GROUPID}%{POSINT:SEVERITYLEVEL}%{DATA:MESSAGECODE}\s%{NOTSPACE}%{GREEDYDATA:MESSAGE}\s

PS1: WIN3,TZ1,PROCESSNAME ARE THE CUSTOM PATTERNS

PS: I HAVE WRITTEN THE PATTERN FOR THESE FOLLOWING TYPES OF LOGS. I DONT HAVE ANY PROBLEM IN WRITING GROK PATTERN FOR THE LOGS IN MY QUEST'N. I GET MANY TYPES OF LOGS FROM MY ROUTER, SO SHOULD I KEEP WRITING GROK PATTERNS OR IS THERE ANY OTHER WAY

5

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:
There's also no need to use all caps in your text.

It looks like most of the patterns share the same common elements. Why not create some custom patterns and leverage those?

6

Sorry for that I will do it from next time
Okay @warkolm so your point is write custom patterns . There is no other way. Am I right??

7

We are working on automated grok discovery with ML. I am not sure of the time line though.

8

Thank you @warkolm

9

This site sometimes comes in handy to get a baseline to work from:
https://grokdebug.herokuapp.com/discover

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.