Filebeat Fortinet have error message

fadjar340 · December 10, 2020, 11:20am

I saw you're the person that give more comment on Filebeat Fortinet module, so I directly ask for help.

I just send my fortinet log into my rsyslog server and save it into the file then I enabled the fortinet modules in Filebeat. I already followed all the procedures to enable the module in this URL .

I have error.message as follow:


    "message": "Dec 10 17:59:54 192.100.254.68 date=2020-12-10 time=17:59:54 devname=\"FW_VPN_Jasindo_DRC\" devid=\"FG3H1E5819900316\" logid=\"0000000013\" type=\"traffic\" subtype=\"forward\" level=\"notice\" vd=\"root\" eventtime=1607597994 srcip=192.168.32.38 srcname=\"VCENTER-DRC-BSD\" srcport=46418 srcintf=\"port2\" srcintfrole=\"lan\" dstip=192.168.203.6 dstport=445 dstintf=\"port1\" dstintfrole=\"wan\" poluuid=\"b67b21c8-9185-51e9-efbb-a32dfe6a402c\" sessionid=2387369696 proto=6 action=\"server-rst\" policyid=3 policytype=\"policy\" service=\"SMB\" dstcountry=\"Reserved\" srccountry=\"Reserved\" trandisp=\"noop\" duration=1 sentbyte=104 rcvdbyte=80 sentpkt=2 rcvdpkt=2 appcat=\"unscanned\" devtype=\"Windows PC\" devcategory=\"Windows Device\" osname=\"Windows 7 / 2008 R2\" unauthuser=\"vcenter-drc-bsd$\" unauthusersource=\"kerberos\" mastersrcmac=\"00:50:56:ae:1a:90\" srcmac=\"00:50:56:ae:1a:90\" srcserver=1",
    "error": {
      "message": "Provided Grok expressions do not match field value: [Dec 10 17:59:54 192.100.254.68 date=2020-12-10 time=17:59:54 devname=\\\"FW_VPN_Jasindo_DRC\\\" devid=\\\"FG3H1E5819900316\\\" logid=\\\"0000000013\\\" type=\\\"traffic\\\" subtype=\\\"forward\\\" level=\\\"notice\\\" vd=\\\"root\\\" eventtime=1607597994 srcip=192.168.32.38 srcname=\\\"VCENTER-DRC-BSD\\\" srcport=46418 srcintf=\\\"port2\\\" srcintfrole=\\\"lan\\\" dstip=192.168.203.6 dstport=445 dstintf=\\\"port1\\\" dstintfrole=\\\"wan\\\" poluuid=\\\"b67b21c8-9185-51e9-efbb-a32dfe6a402c\\\" sessionid=2387369696 proto=6 action=\\\"server-rst\\\" policyid=3 policytype=\\\"policy\\\" service=\\\"SMB\\\" dstcountry=\\\"Reserved\\\" srccountry=\\\"Reserved\\\" trandisp=\\\"noop\\\" duration=1 sentbyte=104 rcvdbyte=80 sentpkt=2 rcvdpkt=2 appcat=\\\"unscanned\\\" devtype=\\\"Windows PC\\\" devcategory=\\\"Windows Device\\\" osname=\\\"Windows 7 / 2008 R2\\\" unauthuser=\\\"vcenter-drc-bsd$\\\" unauthusersource=\\\"kerberos\\\" mastersrcmac=\\\"00:50:56:ae:1a:90\\\" srcmac=\\\"00:50:56:ae:1a:90\\\" srcserver=1]"
    }

Just to check the Grok in the Fortinet pipeline:

{
    "grok": {
      "patterns": [
        "%{SYSLOG5424PRI}%{GREEDYDATA:syslog5424_sd}$"
      ],
      "field": "message"
    }
  }

The fortinet config as follow:

config log syslogd setting
      set status enable
      set server "rsyslog"
      set mode legacy-reliable
      set port 5514
      set facility local7
      set source-ip ''
      set format defaults
end

Also this is my fortinet.yml

# Module: fortinet
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-fortinet.html

- module: fortinet
  firewall:
    enabled: true

    # Set which input to use between tcp, udp (default) or file.
    var.input: file

    # The interface to listen to syslog traffic. Defaults to
    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
    #var.syslog_host: 0.0.0.0

    # The port to listen for syslog traffic. Defaults to 9004.
    #var.syslog_port: 9004

    # Set paths for the log files when file input is used.
    var.paths: ["/datalog/192.100.254.68/*.log"]

    # Toggle output of non-ECS fields (default true).
    var.rsa_fields: false

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    var.tz_offset: +07:00

  clientendpoint:
    enabled: false

    # Set which input to use between udp (default), tcp or file.
    var.input: file
    # var.syslog_host: localhost
    # var.syslog_port: 9510

    # Set paths for the log files when file input is used.
    # var.paths: 

    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: +07:00

  fortimail:
    enabled: false

    # Set which input to use between udp (default), tcp or file.
    var.input: file
    # var.syslog_host: localhost
    # var.syslog_port: 9529

    # Set paths for the log files when file input is used.
    # var.paths:
  
    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: +07:00

  fortimanager:
    enabled: false

    # Set which input to use between udp (default), tcp or file.
    #var.input: file
    # var.syslog_host: localhost
    # var.syslog_port: 9530

    # Set paths for the log files when file input is used.
    # var.paths:
 
    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: +07:00

Is there any workaround to make the fortinet index working?

BTW, I just enable panw.yml and it works flawlesly.
Appreciate your help.

[Edit]
Elasticsearch: 7.10.0
Filebeat: 7.10.0

Regards,
Fadjar340

fadjar340 · December 11, 2020, 4:20am

After do some research, finaly I found the problem.

In the grok processor for log file input, I modified and added this:

%{SYSLOGTIMESTAMP} %{SYSLOGHOST} %{GREEDYDATA:syslog5424_sd}$

The original is:

%{SYSLOG5424PRI}%{GREEDYDATA:syslog5424_sd}$

Perhaps you need some pipe into the grok processor if the input using file as follow:

%{SYSLOG5424PRI}%{GREEDYDATA:syslog5424_sd}$|%{SYSLOGTIMESTAMP} %{SYSLOGHOST} %{GREEDYDATA:syslog5424_sd}$

This grok statement will handle direct input udp/tcp and file as well.

Now, my fortinet logs working as expected.

Regards,
Fadjar340

Sergii_Patsora · December 22, 2020, 12:35pm

Hello,

I have similar problem with ELK 7.10 and FortiOS v6.0.11 - none of these pipelines work for me.

Provided Grok expressions do not match field value: [Dec 22 14:15:18 fg200d date=2020-12-22 time=14:15:17 devname="FG200D4Q26914264" devid="FG200D4Q26914264" logid="1059028704" type="utm" subtype="app-ctrl" eventtype="app-ctrl-all" level="information" vd="root" eventtime=1608639317 appid=16009 srcip=10.10.2.232 dstip=77.75.149.201 srcport=53234 dstport=80 srcintf="port1" srcintfrole="lan" dstintf="port2" dstintfrole="undefined" proto=6 service="HTTP" direction="outgoing" policyid=1 sessionid=380781585 applist="appcontrol_MTU" appcat="Update" app="MS.Windows.Update" action="pass" hostname="2.tlu.dl.delivery.mp.microsoft.com" incidentserialno=1278502712 url="/filestreamingservice/files/3b79c97b-09ea-41fb-92f1-4589f4e2f3c3?P1=1608639938&P2=402&P3=2&P4=OmtIZ1o%2bo5%2bKUyN9lEFm0SjQ4QJ5ebb76H%2fqxg2cJ7iOnOnTCDLJq4yu5i9r%2f9J8yXdoYwZNvQ63qSxQfVqg4A%3d%3d" msg="Update: MS.Windows.Update," apprisk="elevated"]

Can anyone give me any suggestion? Thanks.

fadjar340 · December 22, 2020, 12:57pm

I have document that explain how to change the ingest pipeline of the fortinet.
In my case, I use log file as input, not from UDP/TCP port and all the information was ingested.
If you interested, please message me..

ylasri · December 22, 2020, 2:03pm

Open the following ingest pipeline for fortinet under your filebeat

${FILEBEAT_HOME}/module/fortinet/firewall/ingest/pipeline.yml

And add the new pattern

    - '%{SYSLOGTIMESTAMP}\s*%{SYSLOGHOST}\s*{GREEDYDATA:syslog5424_sd}$'

Final config should looks like

description: Pipeline for parsing fortinet firewall logs
processors:
- set:
    field: event.ingested
    value: '{{_ingest.timestamp}}'
- grok:
    field: message
    patterns:
    - '%{SYSLOGTIMESTAMP}\s*%{SYSLOGHOST}\s*{GREEDYDATA:syslog5424_sd}$'
    - '%{SYSLOG5424PRI}%{GREEDYDATA:syslog5424_sd}$'
- kv:
    field: syslog5424_sd
    field_split: " (?=[a-z\\_\\-]+=)"
    value_split: "="
    prefix: "fortinet.tmp."
    ignore_missing: true
    ignore_failure: false
    trim_value: "\""

fadjar340 · December 22, 2020, 2:12pm

@ylasri, thank you for your snippet.
Some condition also related to the Date when the syslog using timezone not UTC.
In my case, my timezone +07:00, the pipeline save next 7 hours of my localtime.

Below my procedure to change the ingestion:

In Kibana - Stack Management, do some changes of Ingest Node Pipelines - filebeat-7.10.0-fortinet-firewall-pipeline
Edit filebeat-7.10.0-fortinet-firewall-pipeline
Find Grok in the second line below Set, upper Key-value (KV)
Select the Grok, Manage and Edit
Find Patterns
Change the patterns as follow:

%{SYSLOG5424PRI}%{GREEDYDATA:syslog5424_sd}$|(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource}+(?: %{SYSLOGPROG}:|) %{GREEDYDATA:syslog5424_sd}$

Update the Grok processor
Find 2 DATE processor below SET upper GSUB
In the second DATE processor where: Condition (optional) as follow: ctx.fortinet?.firewall?.tz == null, fill the Timezone (optional) with your timezone, ie: +07:00, as time zone of the syslog server. This value will Prevent the timestamp not following your timezone but Next 7 hours (in my case)
Update the Date Processor
Save Pipeline at the bottom of the pipeline page

The Filebeat ingestion pipeline also have error not only Fortinet, also F5 and Checkpoint in my case using log file input using rsyslog. In my experience, Paloalto works flawlessly...

ylasri · December 22, 2020, 2:20pm

I added a new FR to update the module
You can comment with differents example if you wish

Sergii_Patsora · December 22, 2020, 2:34pm

Appreciate your help - now everything works as it should. Thank you all!

system · January 19, 2021, 4:34pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.