Logstash Fortinet Grok Pattern

Elastic Stack Logstash
1

HI All:

I am working on another GROK pattern and hitting some snags. Here are the details:

Sample Log Line:

logver=0702071577 idseq=237867271677022888 itime=1723855010 devid="FGT60E4Q17040542" devname="central-il-cu" vd="root" date=2024-08-16 time=19:36:47 eventtime=1723855007026597042 tz="-0500" logid="0000000020" type="traffic" subtype="forward" level="notice" srcip=1.1.1.1srcport=63018 srcintf="HSwitch" srcintfrole="undefined" dstip=2.2.2.2 dstport=443 dstintf="wan1" dstintfrole="wan" srccountry="Reserved" dstinetsvc="Slack-Slack" dstcountry="United States" dstregion="1830" dstcity="Dublin" dstreputation=5 sessionid=198760625 proto=6 action="accept" policyid=1 policytype="policy" poluuid="30b62a18-cfef-51ed-c877-8881892f8c48" service="Slack-Slack" trandisp="snat" transip=3.3.3.3transport=63018 appid=43345 app="Slack" appcat="Collaboration" apprisk="elevated" applist="block-p2p" duration=15220 sentbyte=231887 rcvdbyte=326742 sentpkt=3389 rcvdpkt=2022 sentdelta=2184 rcvddelta=2287

Here is the GROK Pattern:

This Grok pattern works when I run the Grok Debugger in Elastic.

match => { "message" => "logver=%{NUMBER:log_version}" "idseq=%{NUMBER:idseq}" "itime={NUMBER:itime}" "devid=%{DATA:devid}" "devname=%{DATA:devname}" "vd=%{DATA:vd}" "date=%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}" "time=%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}" "eventtime=%{NUMBER:eventtime}" "tz=%{DATA:tz}" "logid=%{DATA:logid}" "type=%{DATA:type}" "subtype=%{DATA:subtype}" "level=%{DATA:level}" "srcip=%{IP:srcip}" "srcport=%{DATA:srcport}" "srcintf=%{DATA:srcintf}" "srcintfrole=%{DATA:srcintfrole}" "dstip=%{IP:dstip}" "dstport=%{DATA:dstport}" "dstintf=%{DATA:dstintf}" "dstintfrole=%{DATA:dstintfrole}" "srccountry=%{DATA:srccountry}" "dstinetsvc=%{DATA:dstinetsvc}" "dstcountry=%{DATA:dstcountry}" "dstregion=%{DATA:dstregion}" "dstcity=%{DATA:dstcity}" "dstreputation=%{DATA:dstreputation}" "sessionid=%{DATA:sessionid}" "proto=%{DATA:proto}" "action=%{DATA:action}" "policyid=%{DATA:policyid}" "policytype=%{DATA:policytype}" "poluuid=%{DATA:poluuid}" "service=%{DATA:service}" "trandisp=%{DATA:trandisp}" "transip=%{IP:transip}" "transport=%{DATA:transport}" "appid=%{DATA:appid}" "app=%{DATA:app}" "appcat=%{DATA:appcat}" "apprisk=%{DATA:apprisk}" "applist=%{DATA:applist}" "duration=%{DATA:duration}" "sentbyte=%{DATA:sentbyte}" "rcvdbyte=%{DATA:rcvdbyte}" "sentpkt=%{DATA:sentpkt}" "rcvdpkt=%{DATA:rcvdpkt}" "sentdelta=%{DATA:sentdelta}" "rcvddelta=%{DATA:rcvddelta}"}

This is the Error I am getting:

] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 10, column 84 (byte 218) after filter {\r\n  grok {\r\n    match => { \"message\" => \"logver=%{NUMBER:log_version}\" \"idseq=%{NUMBER:idseq}\" ", :backtrace=>[

Can someone please look and let me know what I am missing from the GROK pattern?

Note: I have change the pattern a few times and each time the error is the same.

Thanks
Anthony

2

The compiler is consuming logver=%{NUMBER:log_version}" as the pattern that it needs to match against [message]. Then it is taking "idseq=%{NUMBER:idseq}" as the next field name and expecting to find => to separate it from "itime={NUMBER:itime}".

You could put single quotes around the entire pattern

match => { "message" => '"logver=%{NUMBER:log_version}" ... "rcvddelta=%{DATA:rcvddelta}"' }

or, more likely, remove all of the double quotes, since they do not appear in the log line you are trying to parse.

You can probably replace a lot of those DATA patterns with NOTSPACE, which will fail far faster if the log line does not match. To see what I mean, try deleting logver= from a log line and then sending it through that grok filter.

You might also want to condider using a kv filter instead of grok.

3

I have never used a KV filter. Can you point me in the direction of getting started to look into it as I adjust the GROK.

I tried removing the all of the " and just wrapping the entire line around a " "

I tried wrapping the entire line around ' '

neither work. The ingest runs, but everything gets added to 1 column event.original.

4

The documentation is here. The default separator for keys and values is =, and the default separator for key=value pairs is space, which means you can use the defaults and

kv {}

will produce

     "policyid" => "1",
          "app" => "Slack",
      "service" => "Slack-Slack",
      "srcintf" => "HSwitch",
      "subtype" => "forward",
      "poluuid" => "30b62a18-cfef-51ed-c877-8881892f8c48",

etc. It even silently strips the quotes around field values if they are present. If some of your fields contain spaces it can get a bit more complicated, but cross that bridge when you come to it.

BTW, if you are parsing Fortinet logs in the elastic stack then starting with grok in logstash may well not be the way to do it.

5

Right, you will be getting a _grokparsefailure. Your pattern is very, very complicated, and every field has to match. The way to debug that is to start with

grok { match => { "message" => 'logver=%{NUMBER:log_version} " } }

and test if that works. If it does then add a field to the pattern and test again

grok { match => { "message" => 'logver=%{NUMBER:log_version} idseq=%{NUMBER:idseq} " } }

if that works then add the next field. At that point you will hopefully realize that when you add itime={NUMBER:itime} it stops working, which happens because you are missing a %.

As you keep adding fields you will discover problems like

         "vd" => "\"root\"",
    "devname" => "\"central-il-cu\"",

which means you need to add double quotes around those fields in the pattern (using a kv filter fixes this for you by default).

Keep adding and it will blow up at srcip because you damaged with log line when redacting the IP address. You may also find that having the date and time values split across 6 fields instead of two is not useful, so you need to define custom patterns to consume them.

Using kv{} is a lot simpler.

6

Thanks @Badger KV Filter is my new friend.