Grok parsing, still not solved

Elastic Stack Logstash
1

Hello, I'm still struggling with the syslog parsing issue.

So, my config file looks like this:

input {
udp {
port => 5514
type => syslog
}

tcp {
port => 5514
type => syslog
}
}

filter {

            grok {
                    patterns_dir => ["/etc/logstash/patterns"]
                    match => ["message", "%{SYSLOG5424LINE}"]
                    overwrite => [ "message" ]
                    tag_on_failure => [ "failure_grok_fortigate" ]
            }

            kv { }

            if [msg] {
                    mutate {
                            replace => [ "message", "%{msg}" ]
                    }
            }

            mutate {
                    add_field => ["logTimestamp", "%{date} %{time}"]
                    add_field => ["loglevel", "%{level}"]
                    replace => [ "fortigate_type", "%{type}"]
                    replace => [ "fortigate_subtype", "%{subtype}"]
                    remove_field => [ "msg", "type", "level", "date", "time" ]
            }
            date {
                    locale => "en"
                    match => ["logTimestamp", "YYYY-MM-dd HH:mm:ss"]
                    remove_field => ["logTimestamp", "year", "month", "day", "time", "date"]
                    add_field => ["type", "syslog"]
            }

Then when debug this field is having problems:

"<189>date" => "2019-02-08",
"logTimestamp" => "%{date} 19:00:34",
"message" => "<189>date=2019-02-08 time=19:00:35

So the parsing of the date doesn't go well...

This is an example of a raw log line:

1: date=2019-02-08 time=18:35:20 logid=0000000011 type=traffic subtype=forward level=warning vd=root srcip=3.4.1.1 srcport=45679 srcintf="internal3" dstip=8.8.8.8 dstport=53 dstintf="wan1" poluuid=45245042-ccf9-51e6-28f5-f8fa6131fcdf sessionid=32098621 proto=17 action=dns policyid=1 appcat="unscanned" crscore=5 craction=262144 crlevel=low

Anyone?

2

The grok is doing nothing for you. Just remove it. In the mutate

        add_field => ["logTimestamp", "%{date} %{time}"]

the date field is sdate, not date.

You need to strip the priority off the beginning of the message. I suggest you use

mutate { gsub => [ "message", "^<[0-9]+>", "" ] }

just before the kv.

3

Thanks for aswering! I tried it and parsing still fails though..

"logTimestamp" => "%{sdate} 13:15:13",

However, message field is fine now:

"message" => "date=2019-02-09 time=13:17:19

4

Anything?

5

Well, if you have changed the message format to use date rather than sdate then you should use date instead of sdate in the mutate.

6

Like this? :

filter {

            kv { }

            if [msg] {
                    mutate {
                            replace => [ "message", "%{msg}" ]
                    }
            }

            mutate {
                    add_field => ["logTimestamp", "%{date} %{time}"]
                    add_field => ["loglevel", "%{level}"]
                    replace => [ "fortigate_type", "%{type}"]
                    replace => [ "fortigate_subtype", "%{subtype}"]
                    remove_field => [ "msg", "type", "level", "date", "time" ]
                    gsub => [ "message", "^<[0-9]+>", "" ]
            }
            date {
                    locale => "en"
                    match => ["logTimestamp", "yyyy-MM-dd'T'HH:mm:ss"]
                    remove_field => ["logTimestamp", "year", "month", "day", "time", "date"]
                    add_field => ["type", "syslog"]
            }
7

The date format has to match the format of the date. You are saying that you have

"message" => "date=2019-02-09 time=13:17:19

and you add a field that links those two with a space

add_field => ["logTimestamp", "%{date} %{time}"]

and yet you try to parse it as though they were linked with 'T'

match => ["logTimestamp", "yyyy-MM-dd'T'HH:mm:ss"]

You have to use the same format in add_field and the date filter.

8

Like this?
match => ["logTimestamp", "yyyy-MM-dd HH:mm:ss"]

9

Yes.

10

Hmm, it's still not parsed..
"logTimestamp" => "%{date} 14:29:01",

Should I use extra double quotes with %{date} and %{time} fields?

11

What does a complete event look like when you use this output?...

output { stdout { codec => rubydebug } }
12

This is the complete output when I start with stdout:

{
"sentpkt" => "1",
"<189>date" => "2019-02-12",
"vd" => "root",
"subtype" => "forward",
"fortigate_type" => "traffic",
"dstintf" => "wan1",
"trandisp" => "snat",
"devid" => "FGT67C3913456462",
"poluuid" => "45245042-ccf9-51e6-28f5-f8fa6131fcdf",
"host" => "192.168.1.25",
"transport" => "40212",
"tags" => [
[0] "_dateparsefailure"
],
"srcip" => "192.168.1.15",
"logTimestamp" => "%{date} 14:29:01",
"logid" => "0000000013",
"rcvdpkt" => "1",
"@version" => "1",
"dstport" => "1087",
"service" => "tcp/1087",
"srcintf" => "internal3",
"duration" => "5",
"transip" => "18.17.2.4",
"sentbyte" => "60",
"message" => "date=2019-02-12 time=14:29:01 devname=FW01 devid=FGT40C6787012462 logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip=192.168.1.15 srcport=40212 srcintf="internal3" dstip=19.12.18.16 dstport=1087 dstintf="wan1" poluuid=45245042-ccf9-51e6-28f5-f8fa6131fcdf sessionid=32556980 proto=6 action=close policyid=1 dstcountry="Chile" srccountry="Reserved" trandisp=snat transip=18.17.2.4 transport=40212 service="tcp/1087" duration=5 sentbyte=60 rcvdbyte=40 sentpkt=1 rcvdpkt=1 appcat="unscanned"",
"fortigate_subtype" => "forward",
"loglevel" => "notice",
"devname" => "FW01",
"dstip" => "19.12.18.16",
"appcat" => "unscanned",
"policyid" => "1",
"sessionid" => "32556980",
"@timestamp" => 2019-02-12T13:27:30.414Z,
"rcvdbyte" => "40",
"proto" => "6",
"action" => "close",
"dstcountry" => "Chile",
"srcport" => "40212",
"srccountry" => "Reserved"
}

13

OK, so you do not have a field called date. You have a field called

"<189>date" => "2019-02-12"

As I said 4 days ago... You need to strip the priority off the beginning of the message. I suggest you use

mutate { gsub => [ "message", "^<[0-9]+>", "" ] }

just before the kv.

1 Like
14

Ah Before the KV, now it makes sense.
So the priority hasn't been stripped off after all..?

15

Thanks, it's all sorted. Kudos to you!

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.