# Grok parsing, still not solved **URL:** \ **Category:** Logstash\ **Created:** [February 8, 2019, 6:20pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676 "2019-02-08T18:20:14Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![heskez](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@heskez](https://discuss.elastic.co/u/heskez)\ **Post date:** [February 8, 2019, 6:20pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/1 "2019-02-08T18:20:14Z")
Hello, I'm still struggling with the syslog parsing issue. So, my config file looks like this: input { udp { port =\> 5514 type =\> syslog } tcp { port =\> 5514 type =\> syslog } } filter { ``` grok { patterns_dir => ["/etc/logstash/patterns"] match => ["message", "%{SYSLOG5424LINE}"] overwrite => ["message"] tag_on_failure => ["failure_grok_fortigate"] } kv { } if [msg] { mutate { replace => ["message", "%{msg}"] } } mutate { add_field => ["logTimestamp", "%{date} %{time}"] add_field => ["loglevel", "%{level}"] replace => ["fortigate_type", "%{type}"] replace => ["fortigate_subtype", "%{subtype}"] remove_field => ["msg", "type", "level", "date", "time"] } date { locale => "en" match => ["logTimestamp", "YYYY-MM-dd HH:mm:ss"] remove_field => ["logTimestamp", "year", "month", "day", "time", "date"] add_field => ["type", "syslog"] } ``` Then when debug this field is having problems: "\<189\>date" =\> "2019-02-08", **"logTimestamp" =\> "%{date} 19:00:34",** "message" =\> "\<189\>date=2019-02-08 time=19:00:35 So the parsing of the date doesn't go well... This is an example of a raw log line: 1: date=2019-02-08 time=18:35:20 logid=0000000011 type=traffic subtype=forward level=warning vd=root srcip=3.4.1.1 srcport=45679 srcintf="internal3" dstip=8.8.8.8 dstport=53 dstintf="wan1" poluuid=45245042-ccf9-51e6-28f5-f8fa6131fcdf sessionid=32098621 proto=17 action=dns policyid=1 appcat="unscanned" crscore=5 craction=262144 crlevel=low Anyone? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 8, 2019, 6:34pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/2 "2019-02-08T18:34:41Z")
The grok is doing nothing for you. Just remove it. In the mutate ``` add_field => ["logTimestamp", "%{date} %{time}"] ``` the date field is sdate, not date. You need to strip the priority off the beginning of the message. I suggest you use ``` mutate { gsub => ["message", "^<[0-9]+>", "" ] } ``` just before the kv. ---
**Author:** ![heskez](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@heskez](https://discuss.elastic.co/u/heskez)\ **Post date:** [February 9, 2019, 12:19pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/3 "2019-02-09T12:19:14Z")
Thanks for aswering! I tried it and parsing still fails though.. "logTimestamp" =\> "%{sdate} 13:15:13", However, message field is fine now: "message" =\> "date=2019-02-09 time=13:17:19 ---
**Author:** ![heskez](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@heskez](https://discuss.elastic.co/u/heskez)\ **Post date:** [February 12, 2019, 9:40am UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/4 "2019-02-12T09:40:51Z")
Anything? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 12, 2019, 11:59am UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/5 "2019-02-12T11:59:38Z")
Well, if you have changed the message format to use date rather than sdate then you should use date instead of sdate in the mutate. ---
**Author:** ![heskez](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@heskez](https://discuss.elastic.co/u/heskez)\ **Post date:** [February 12, 2019, 12:14pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/6 "2019-02-12T12:14:14Z")
Like this? : filter { ``` kv { } if [msg] { mutate { replace => ["message", "%{msg}"] } } mutate { add_field => ["logTimestamp", "%{date} %{time}"] add_field => ["loglevel", "%{level}"] replace => ["fortigate_type", "%{type}"] replace => ["fortigate_subtype", "%{subtype}"] remove_field => ["msg", "type", "level", "date", "time"] gsub => ["message", "^<[0-9]+>", "" ] } date { locale => "en" match => ["logTimestamp", "yyyy-MM-dd'T'HH:mm:ss"] remove_field => ["logTimestamp", "year", "month", "day", "time", "date"] add_field => ["type", "syslog"] } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 12, 2019, 12:53pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/7 "2019-02-12T12:53:35Z")
The date format has to match the format of the date. You are saying that you have ``` "message" => "date=2019-02-09 time=13:17:19 ``` and you add a field that links those two with a space ``` add_field => ["logTimestamp", "%{date} %{time}"] ``` and yet you try to parse it as though they were linked with 'T' ``` match => ["logTimestamp", "yyyy-MM-dd'T'HH:mm:ss"] ``` You have to use the same format in add\_field and the date filter. ---
**Author:** ![heskez](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@heskez](https://discuss.elastic.co/u/heskez)\ **Post date:** [February 12, 2019, 1:03pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/8 "2019-02-12T13:03:50Z")
> [@Badger](#): > > match =\> ["logTimestamp", "yyyy-MM-dd'T'HH:mm:ss"] Like this? match =\> ["logTimestamp", "yyyy-MM-dd HH:mm:ss"] ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 12, 2019, 1:33pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/9 "2019-02-12T13:33:39Z")
> [@heskez](#): > > yyyy-MM-dd HH:mm:ss Yes. ---
**Author:** ![heskez](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@heskez](https://discuss.elastic.co/u/heskez)\ **Post date:** [February 12, 2019, 1:38pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/10 "2019-02-12T13:38:52Z")
Hmm, it's still not parsed.. "logTimestamp" =\> "%{date} 14:29:01", Should I use extra double quotes with %{date} and %{time} fields? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 12, 2019, 1:45pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/11 "2019-02-12T13:45:00Z")
What does a complete event look like when you use this output?... ``` output { stdout { codec => rubydebug } } ``` ---
**Author:** ![heskez](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@heskez](https://discuss.elastic.co/u/heskez)\ **Post date:** [February 12, 2019, 2:01pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/12 "2019-02-12T14:01:55Z")
This is the complete output when I start with stdout: { "sentpkt" =\> "1", "\<189\>date" =\> "2019-02-12", "vd" =\> "root", "subtype" =\> "forward", "fortigate\_type" =\> "traffic", "dstintf" =\> "wan1", "trandisp" =\> "snat", "devid" =\> "FGT67C3913456462", "poluuid" =\> "45245042-ccf9-51e6-28f5-f8fa6131fcdf", "host" =\> "192.168.1.25", "transport" =\> "40212", "tags" =\> [ [0] "\_dateparsefailure" ], "srcip" =\> "192.168.1.15", "logTimestamp" =\> "%{date} 14:29:01", "logid" =\> "0000000013", "rcvdpkt" =\> "1", "@version" =\> "1", "dstport" =\> "1087", "service" =\> "tcp/1087", "srcintf" =\> "internal3", "duration" =\> "5", "transip" =\> "18.17.2.4", "sentbyte" =\> "60", "message" =\> "date=2019-02-12 time=14:29:01 devname=FW01 devid=FGT40C6787012462 logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip=192.168.1.15 srcport=40212 srcintf="internal3" dstip=19.12.18.16 dstport=1087 dstintf="wan1" poluuid=45245042-ccf9-51e6-28f5-f8fa6131fcdf sessionid=32556980 proto=6 action=close policyid=1 dstcountry="Chile" srccountry="Reserved" trandisp=snat transip=18.17.2.4 transport=40212 service="tcp/1087" duration=5 sentbyte=60 rcvdbyte=40 sentpkt=1 rcvdpkt=1 appcat="unscanned"", "fortigate\_subtype" =\> "forward", "loglevel" =\> "notice", "devname" =\> "FW01", "dstip" =\> "19.12.18.16", "appcat" =\> "unscanned", "policyid" =\> "1", "sessionid" =\> "32556980", "@timestamp" =\> 2019-02-12T13:27:30.414Z, "rcvdbyte" =\> "40", "proto" =\> "6", "action" =\> "close", "dstcountry" =\> "Chile", "srcport" =\> "40212", "srccountry" =\> "Reserved" } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 12, 2019, 2:04pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/13 "2019-02-12T14:04:13Z")
> [@Badger](#): > > mutate { gsub =\> ["message", "^\<[0-9]+\>", "" ] } OK, so you do not have a field called date. You have a field called ``` "<189>date" => "2019-02-12" ``` As I said 4 days ago... You need to strip the priority off the beginning of the message. I suggest you use ```auto mutate { gsub => ["message", "^<[0-9]+>", "" ] } ``` just before the kv. ---
**Author:** ![heskez](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@heskez](https://discuss.elastic.co/u/heskez)\ **Post date:** [February 12, 2019, 3:04pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/14 "2019-02-12T15:04:57Z")
Ah _Before_ the KV, now it makes sense. So the priority hasn't been stripped off after all..? ---
**Author:** ![heskez](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@heskez](https://discuss.elastic.co/u/heskez)\ **Post date:** [February 13, 2019, 1:37pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/15 "2019-02-13T13:37:46Z")
Thanks, it's all sorted. Kudos to you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 13, 2019, 1:37pm UTC](https://discuss.elastic.co/t/grok-parsing-still-not-solved/167676/16 "2019-03-13T13:37:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.