Grok filter

Elastic Stack Logstash
1

Hi , I am trying to construct grok filter to parse data (tomcat log ) but having issues .

I am using input :

[2017-05-09 16:27:53,156] :|: INFO :|: dfprdsndl22.df.jabodo.com :|: 6771ebd6eb814dc28e479908d542b6f6 :|: [BT:FF, BV:37, BL:en, CC:IN] :|: 103.42.89.250 :|: http://download.filmfanatic.com/index.jhtml?partner=Z1xdm961&theme=01ab12212016&s2=-5658453153796672849&s1=618500 :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: DLPInfo, IP: 103.42.89.250]
[2017-05-09 16:27:53,409] :|: INFO :|: dfprdsndl22.df.jabodo.com :|: 6771ebd6eb814dc28e479908d542b6f6 :|: [BT:FF, BV:37, BL:en, CC:IN] :|: 103.42.89.250 :|: http://download.filmfanatic.com/index.jhtml?partner=Z1xdm961&theme=01ab12212016&s2=-5658453153796672849&s1=618500 :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: SplashPageServed, IP: 103.42.89.250]
[2017-05-09 16:27:53,513] :|: INFO :|: dfprdsndl22.df.jabodo.com :|: :|: [BT:FF, BV:53, BL:en, CC:US] :|: 96.236.136.65 :|: http://puzzlegamesdaily.dl.myway.com/blank.jhtml :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: BlankPageServed, IP: 96.236.136.65]
[2017-05-09 16:27:53,523] :|: INFO :|: dfprdsndl22.df.jabodo.com :|: f60bd13c2c154325b35edfa737166a15 :|: [BT:CHROME, BV:53, BL:en, CC:US] :|: 169.241.55.127 :|: :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: ToolbarDetect, IP: 169.241.55.127]
[2017-05-09 16:27:54,011] :|: INFO :|: dfprdsndl22.df.jabodo.com :|: :|: [BT:CHROME, BV:57, BL:en, CC:US] :|: 98.93.68.71 :|: :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: PageView, IP: 98.93.68.71]

Trying to get parse data in the format of

timestamp :|: level :|: hostname :|: requestid :|:browser key :|: requestip :|: url :|: client ip

What grok filter will be helpful for this ?

Thanks,

Nikhil

2

Current filter nt working
filter {
if [type] == "syslog" {
mutate {
gsub => ["message",":|:"," "]
}
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog
_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

3

Well, a syslog grok expression won't work for non-syslog inputs. Have a look at the grok constructor web site for pointers.

4

Ok ..I have modified and restarted it . Still not seeing correct output on kibana
filter {
if [type] == "log" {
mutate {
gsub => ["message",":|:"," "]
}
grok {
match => { "message" => "%{DATESTAMP:timestamp} %{WORD:severity} %{HOST:hostname} %{JAVACLASS:class} %{IP:clientip} %{GREEDYDATA:mess
age}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "log_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

5

Show us what you get. Copy/paste from the JSON tab in Kibana's discover view. No screenshots.

6

works with
(?[[^]]])%{SPACE}:|:%{SPACE}%{WORD:level}%{SPACE}:|:%{SPACE}%{USERNAME:hostname}%{SPACE}:|:%{SPACE}%{GREEDYDATA:coidkey}%{SPACE}:|:%{SPACE}%{GREEDYDATA:clientinfo}%{SPACE}:|:%{SPACE}%{IP:clientIP}%{SPACE}:|:%{SPACE}%{GREEDYDATA:Url}%{SPACE}:|:%{SPACE}%{JAVACLASS:class}%{SPACE}:|:%{SPACE}(?[[^]]])

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.