# Grok in realtime fails but in Debuuger works happy
**URL:** https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552
**Category:** Logstash
**Created:** [May 12, 2015, 12:41pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552 "2015-05-12T12:41:26Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@tassa](https://discuss.elastic.co/u/tassa)
#### Post date: [May 12, 2015, 12:41pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/1 "2015-05-12T12:41:26Z")
Hello Everyone
im trying to parse my fotigate firewal logs with Grok Plese help
im getting this failure message "\_grokparsefailure\_sysloginput" (kibana gui) but Grok debuggr works fine
im using logstash 1.5.0-rc3
java version "1.7.0\_65"
Any help will appriciate
filter{
if [type] == "syslog" {
grok {
match =\> [
'message', "._date=(?:%{YEAR:fw\_year})-(?:%{MONTHNUM:fw\_month})-(?:%{MONTHDAY:fw\_day})_.(?:%{NOTSPACE})_.devname(?:%{NOTSPACE:fw\_name})_.devid=(?:%{NOTSPACE:serialid})_.(?:%{NOTSPACE})_.type=(?:%{NOTSPACE:utm})_.subtype=(?:%{NOTSPACE:IPS})_.eventtype=(?:%{NOTSPACE:signature})_.level=(?:%{NOTSPACE:ips-alert})_.(?:%{NOTSPACE})_.severity=(?:%{NOTSPACE:severity})_.srcip=(?:%{NOTSPACE:srcip})_.dstip=(?:%{NOTSPACE:dstip})_.(?:%{NOTSPACE})_.(?:%{NOTSPACE})_.(?:%{NOTSPACE})_.(?:%{NOTSPACE})_.(?:%{NOTSPACE})_.status=(?:%{NOTSPACE:status})_.proto=(?:%{NOTSPACE:protocol})_.service=(?:%{NOTSPACE:service})_.(?:%{NOTSPACE})_.attackname(?:%{NOTSPACE:attack})_.(?:%{NOTSPACE})_.dstport=(?:%{NOTSPACE:dstport})_.attackid=(?:%{NOTSPACE:attackid})_.sensor=(?:%{NOTSPACE:sensor})_.ref=(?:%{NOTSPACE:reffrenceurl})\*.(?:%{GREEDYDA
TA})"i
]
}
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 12, 2015, 2:12pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/2 "2015-05-12T14:12:44Z")
What do the messages you're trying to parse look like?
---
### Author:  [@tassa](https://discuss.elastic.co/u/tassa)
#### Post date: [May 12, 2015, 2:30pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/3 "2015-05-12T14:30:13Z")
\*This is one of the messages that comes from my fortigate firewall \*
_\<157\>date=2015-05-11 time=10:08:16 devname=RUMOFW01 devid=FG100D3G14823541
logid=0317013312 type=utm subtype=webfilter eventtype=ftgd\_allow
level=notice vd="vpn-s2s" policyid=8893 identidx=0 sessionid=399023432
srcip=172.18.2.28 srcport=50123 srcintf="port7" dstip=87.251.132.211
dstport=80 dstintf="wan1" service="http" hostname="[gadgets.live.com](http://gadgets.live.com)
[http://gadgets.live.com](http://gadgets.live.com)" profiletype="Webfilter\_Profile"
profile="ECI-WEB-Filter-All" status="passthrough" reqtype="direct"
url="/configW7.xml" sentbyte=526 rcvdbyte=419 msg="URL belongs to an
allowed category in policy" method=domain class=0 cat=41 catdesc="Search
Engines and Portals"cvdbyte=419 msg="URL belongs to an allowed category in
policy" method=domain class=0 cat=41 catdesc="Search Engines_
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 12, 2015, 2:47pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/4 "2015-05-12T14:47:58Z")
The grok filter isn't always the best choice. In this case use the [kv filter](http://logstash.net/docs/latest/filters/kv).
```
filter {
grok {
match => [
"message",
"<%{INT:syslog_pri}>%{GREEDYDATA:message}"
]
overwrite => "message"
}
kv { }
}
```
---
### Author:  [@tassa](https://discuss.elastic.co/u/tassa)
#### Post date: [May 12, 2015, 2:50pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/5 "2015-05-12T14:50:59Z")
Okay thanks i wiil try it and update you
Tal
---
### Author:  [@tassa](https://discuss.elastic.co/u/tassa)
#### Post date: [May 12, 2015, 3:04pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/6 "2015-05-12T15:04:19Z")
im sorry id didn't change im still getting "\_grokparsefailure\_sysloginput"
and all the fields are still in the "message" not sperated
These are all my conf files
input.conf output.conf and filter.conf
input {
syslog {
host =\> "x.x.x.x.x"
port =\> 514
codec =\> plain {
charset =\> "ISO-8859-1" }
}
}
output {
elasticsearch {
protocol =\> "http"
}
}
filter {
grok {
match =\> [
"message",
"\<%{INT:syslog\_pri}\>%{GREEDYDATA:message}"
]
overwrite =\> "message"
}
kv { }
}
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 12, 2015, 3:41pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/7 "2015-05-12T15:41:48Z")
Oh, now I get it. You get a `_grokparsefailure_sysloginput` tag, indicating that it's the syslog input's use of grok that's failing:
>
The subsequent grok and kv filter should still be successful though.
---
### Author:  [@tassa](https://discuss.elastic.co/u/tassa)
#### Post date: [May 12, 2015, 8:18pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/8 "2015-05-12T20:18:26Z")
Thank you but not sure i follow you
how exectly to i do it ? can you show me please !
Tal
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 13, 2015, 6:01am UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/9 "2015-05-13T06:01:25Z")
You can't prevent the initial addition of the `_grokparsefailure_sysloginput` tag (but you can remove it afterwards). As I said your grok and kv filters should be working and if they really don't I suggest you scale down your configuration to a minimal example that we can use for debugging. For example, drop inputs and output replace with this:
```
input {
stdin { }
}
output {
stdout {
codec => rubydebug
}
}
```
Then feed the string you think you're getting from syslog to Logstash. The results of that should make it easier to figure out what's going on and decide on a next step.
---
### Author:  [@tassa](https://discuss.elastic.co/u/tassa)
#### Post date: [May 13, 2015, 7:45am UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/10 "2015-05-13T07:45:37Z")
Hi
i have installed the suggested plugin
bin/plugin install logstash-input-syslog
But i still get the nusty message
"tags \_grokparsefailure\_sysloginput"
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 13, 2015, 7:53am UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/11 "2015-05-13T07:53:20Z")
If the plugin wasn't installed you'd have bigger problems.
I repeat: You can't prevent the initial addition of the `_grokparsefailure_sysloginput` tag. Focus on the real problem instead. I have described the steps I think you should take to debug why your grok and kv filters aren't working.
---
### Author:  [@tassa](https://discuss.elastic.co/u/tassa)
#### Post date: [May 13, 2015, 8:07am UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/12 "2015-05-13T08:07:39Z")
Do you have suggestion how to debug this ?
i see no errors in the logstash.log ?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 13, 2015, 10:02am UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/13 "2015-05-13T10:02:16Z")
I gave you some suggestions in my last response ("I suggest you scale down your configuration ...").
---
### Author:  [@tassa](https://discuss.elastic.co/u/tassa)
#### Post date: [May 19, 2015, 2:31pm UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/14 "2015-05-19T14:31:42Z")
Hi Magnus Bäck
After a week of debugging i think i have a lead to find out the root of grok failure
when i use "file input" The grok is parsing the "message" with no failure and i can see all fields correctly
this input works perfect !!
file {
type =\> "fortigate"
path =\> ["/var/log/\*.log"]
}
}
But when i use logstash input syslog
i get grokfailure
The message apperas in the kibana with an ascii character like this
\<157\>date=2015-05-19 time=10:04:32 devname=USFLFW01-MASTER
i have reason to believe that syslog message is been distorted somehow
input {
syslog {
host =\> "x.x.x.x"
port =\> 514
}
}
any advice will be appreciated
Tia
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 20, 2015, 5:58am UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/15 "2015-05-20T05:58:30Z")
The leading \<157\> is expected; see [http://stackoverflow.com/a/30143229/414355](http://stackoverflow.com/a/30143229/414355) for an explanation.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:39am UTC](https://discuss.elastic.co/t/grok-in-realtime-fails-but-in-debuuger-works-happy/552/16 "2017-07-06T05:39:37Z")