Grok pattern not being applied in logstash but working on grok debugger


(Swaroop Chandre) #1

My grok pattern is working on grok debugger but when reading from filebeat my grok pattern is not applied.

My syslog-filter.conf file

filter {
if [type] == "log" {
grok {
patterns_dir => ["/etc/logstash/conf.d/pattern.conf"]
match => { "message" =>"%{SYSLOGTIMESTAMP:date} %{IPV6:sourceip} %{POSINT:seqnum1}: %{POSINT:seqnum2}: %{DATA:date1} %%{DATA:message}-%{POSINT:severity}-%{DATA:mnemonic}: %{GREEDYDATA:log_message}" }
}

}
}

I get output as:

{
"@timestamp": "2017-08-17T07:43:10.892Z",
"beat": {
"hostname": "autosysrv107",
"name": "autosysrv107",
"version": "5.5.1"
},
"input_type": "log",
"message": "Jul 16 07:01:04 2405:200:204:101:172:26:161:172 990708: 990653: Jul 16 07:02:24.749 IST: %TCP-6-BADAUTH: No MD5 digest from 172.16.32.120(646) to 172.26.161.172(23750) tableid - 0",
"offset": 9774736,
"source": "/var/log/messages-20170723",
"type": "log"
}{
"@timestamp": "2017-08-17T07:43:10.892Z",
"beat": {
"hostname": "autosysrv107",
"name": "autosysrv107",
"version": "5.5.1"
},
"input_type": "log",
"message": "Jul 16 07:01:04 2405:200:204:101:172:22:2:96 253818: 253555: Jul 16 07:02:24.827 IST: %TCP-6-BADAUTH: No MD5 digest from 2405:200:201:101:172:22:9:190(179) to 2405:200:201:101:172:22:2:96(12530) (RST) tableid - 0",
"offset": 9774949,
"source": "/var/log/messages-20170723",
"type": "log"
}

Where as I want messages in json as (which I get in grok debugger but failing while reading from filebeat)

{
"date": [
[
"Jul 16 07:01:04"
]
],
"MONTH": [
[
"Jul"
]
],
"MONTHDAY": [
[
"16"
]
],
"TIME": [
[
"07:01:04"
]
],
"HOUR": [
[
"07"
]
],
"MINUTE": [
[
"01"
]
],
"SECOND": [
[
"04"
]
],
"sourceip": [
[
"2405:200:204:101:172:26:161:227"
]
],
"seqnum1": [
[
"1232887"
]
],
"seqnum2": [
[
"1232772"
]
],
"date1": [
[
"Jul 16 07:02:25.386 IST:"
]
],
"message": [
[
"TCP"
]
],
"severity": [
[
"6"
]
],
"mnemonic": [
[
"BADAUTH"
]
],
"log_message": [
[
"No MD5 digest from 172.22.4.146(646) to 172.26.161.227(52241) tableid - 0"
]
]
}


(Magnus Bäck) #2

It looks like the grok filter is never run at all and it's not clear why. Is that grok filter the only filter you have?


(Swaroop Chandre) #3

Sorry for late reply. This resolved.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.