Grok parsing, still not solved

(Erik Heskes) #1

Hello, I'm still struggling with the syslog parsing issue.

So, my config file looks like this:

input {
udp {
port => 5514
type => syslog
}

tcp {
port => 5514
type => syslog
}
}

filter {

            grok {
                    patterns_dir => ["/etc/logstash/patterns"]
                    match => ["message", "%{SYSLOG5424LINE}"]
                    overwrite => [ "message" ]
                    tag_on_failure => [ "failure_grok_fortigate" ]
            }

            kv { }

            if [msg] {
                    mutate {
                            replace => [ "message", "%{msg}" ]
                    }
            }

            mutate {
                    add_field => ["logTimestamp", "%{date} %{time}"]
                    add_field => ["loglevel", "%{level}"]
                    replace => [ "fortigate_type", "%{type}"]
                    replace => [ "fortigate_subtype", "%{subtype}"]
                    remove_field => [ "msg", "type", "level", "date", "time" ]
            }
            date {
                    locale => "en"
                    match => ["logTimestamp", "YYYY-MM-dd HH:mm:ss"]
                    remove_field => ["logTimestamp", "year", "month", "day", "time", "date"]
                    add_field => ["type", "syslog"]
            }

Then when debug this field is having problems:

"<189>date" => "2019-02-08",
"logTimestamp" => "%{date} 19:00:34",
"message" => "<189>date=2019-02-08 time=19:00:35

So the parsing of the date doesn't go well...

This is an example of a raw log line:

1: date=2019-02-08 time=18:35:20 logid=0000000011 type=traffic subtype=forward level=warning vd=root srcip=3.4.1.1 srcport=45679 srcintf="internal3" dstip=8.8.8.8 dstport=53 dstintf="wan1" poluuid=45245042-ccf9-51e6-28f5-f8fa6131fcdf sessionid=32098621 proto=17 action=dns policyid=1 appcat="unscanned" crscore=5 craction=262144 crlevel=low

Anyone?

#2

The grok is doing nothing for you. Just remove it. In the mutate

        add_field => ["logTimestamp", "%{date} %{time}"]

the date field is sdate, not date.

You need to strip the priority off the beginning of the message. I suggest you use

mutate { gsub => [ "message", "^<[0-9]+>", "" ] }

just before the kv.

(Erik Heskes) #3

Thanks for aswering! I tried it and parsing still fails though..

"logTimestamp" => "%{sdate} 13:15:13",

However, message field is fine now:

"message" => "date=2019-02-09 time=13:17:19

(Erik Heskes) #4

Anything?

#5

Well, if you have changed the message format to use date rather than sdate then you should use date instead of sdate in the mutate.

(Erik Heskes) #6

Like this? :

filter {

            kv { }

            if [msg] {
                    mutate {
                            replace => [ "message", "%{msg}" ]
                    }
            }

            mutate {
                    add_field => ["logTimestamp", "%{date} %{time}"]
                    add_field => ["loglevel", "%{level}"]
                    replace => [ "fortigate_type", "%{type}"]
                    replace => [ "fortigate_subtype", "%{subtype}"]
                    remove_field => [ "msg", "type", "level", "date", "time" ]
                    gsub => [ "message", "^<[0-9]+>", "" ]
            }
            date {
                    locale => "en"
                    match => ["logTimestamp", "yyyy-MM-dd'T'HH:mm:ss"]
                    remove_field => ["logTimestamp", "year", "month", "day", "time", "date"]
                    add_field => ["type", "syslog"]
            }
#7

The date format has to match the format of the date. You are saying that you have

"message" => "date=2019-02-09 time=13:17:19

and you add a field that links those two with a space

add_field => ["logTimestamp", "%{date} %{time}"]

and yet you try to parse it as though they were linked with 'T'

match => ["logTimestamp", "yyyy-MM-dd'T'HH:mm:ss"]

You have to use the same format in add_field and the date filter.

(Erik Heskes) #8

Like this?
match => ["logTimestamp", "yyyy-MM-dd HH:mm:ss"]

#9

Yes.

(Erik Heskes) #10

Hmm, it's still not parsed..
"logTimestamp" => "%{date} 14:29:01",

Should I use extra double quotes with %{date} and %{time} fields?

#11

What does a complete event look like when you use this output?...

output { stdout { codec => rubydebug } }
(Erik Heskes) #12

This is the complete output when I start with stdout:

{
"sentpkt" => "1",
"<189>date" => "2019-02-12",
"vd" => "root",
"subtype" => "forward",
"fortigate_type" => "traffic",
"dstintf" => "wan1",
"trandisp" => "snat",
"devid" => "FGT67C3913456462",
"poluuid" => "45245042-ccf9-51e6-28f5-f8fa6131fcdf",
"host" => "192.168.1.25",
"transport" => "40212",
"tags" => [
[0] "_dateparsefailure"
],
"srcip" => "192.168.1.15",
"logTimestamp" => "%{date} 14:29:01",
"logid" => "0000000013",
"rcvdpkt" => "1",
"@version" => "1",
"dstport" => "1087",
"service" => "tcp/1087",
"srcintf" => "internal3",
"duration" => "5",
"transip" => "18.17.2.4",
"sentbyte" => "60",
"message" => "date=2019-02-12 time=14:29:01 devname=FW01 devid=FGT40C6787012462 logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip=192.168.1.15 srcport=40212 srcintf="internal3" dstip=19.12.18.16 dstport=1087 dstintf="wan1" poluuid=45245042-ccf9-51e6-28f5-f8fa6131fcdf sessionid=32556980 proto=6 action=close policyid=1 dstcountry="Chile" srccountry="Reserved" trandisp=snat transip=18.17.2.4 transport=40212 service="tcp/1087" duration=5 sentbyte=60 rcvdbyte=40 sentpkt=1 rcvdpkt=1 appcat="unscanned"",
"fortigate_subtype" => "forward",
"loglevel" => "notice",
"devname" => "FW01",
"dstip" => "19.12.18.16",
"appcat" => "unscanned",
"policyid" => "1",
"sessionid" => "32556980",
"@timestamp" => 2019-02-12T13:27:30.414Z,
"rcvdbyte" => "40",
"proto" => "6",
"action" => "close",
"dstcountry" => "Chile",
"srcport" => "40212",
"srccountry" => "Reserved"
}

#13

OK, so you do not have a field called date. You have a field called

"<189>date" => "2019-02-12"

As I said 4 days ago... You need to strip the priority off the beginning of the message. I suggest you use

mutate { gsub => [ "message", "^<[0-9]+>", "" ] }

just before the kv.

1 Like
(Erik Heskes) #14

Ah Before the KV, now it makes sense.
So the priority hasn't been stripped off after all..?

(Erik Heskes) #15

Thanks, it's all sorted. Kudos to you!

(system) closed #16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.