FILEBEAT module Fortinet : Provided Grok expressions do not match field value

Alex_Lum · April 13, 2021, 9:44am

Hello,
I send fortigate logs via rsylog on /var/log/fortigate* files.

I 've configured filebeat with fortinet module.
On kibana, i get this : Provided Grok expressions do not match field value

It was working with udp 9004 input, but not with file input.

Filebeat.yml is not modified.

module.d/fortinet.yml :

- module: fortinet
  firewall:
    enabled: true
    var.input: file
    var.paths:
       - /var/log/fortigate*

    # var.syslog_host: 0.0.0.0
    # var.syslog_port: 9004

I followed this : Fileabeat grok fail "Provided Grok expressions do not match field value"
./filebeat setup --pipelines -modules="fortinet"
deleted indices and index patterns, but still no luck.

Can you help me?

Thanks.

Marius_Iversen · April 14, 2021, 12:39am

Hello @Alex_Lum

Just to confirm, when you mention that it did work with UDP, do you mean that you sent data directly from fortigate to filebeat UDP input, or that fortinet is sending to rsyslog first?

Is there any reason why the data needs to enter rsyslog first? For example you want to send it to a second location for backup purposes?

Most likely, once the data is received by rsyslog, and it writes it to a file, it modifies the format slightly, would you be able to provide a sample logline from the file?

For reference, this is how the module expects the logfile to look like: beats/fortinet.log at master · elastic/beats · GitHub

Alex_Lum · April 14, 2021, 7:01am

Hello @Marius_Iversen ,

Previously, i sent logs from fortigate to UDP port 9004 and to syslog.
I have to communicate logs with TLS, so i configured forti syslog like this.

config log syslogd setting
    set status enable
    set server "xxxxx"
    set mode reliable
    set port 6514
    set facility local0
    set enc-algorithm high-medium
    set ssl-min-proto-version TLSv1-2
    set certificate "syslog"
    set source-ip "xxxx"
end

I 'd like to send logs TCP with TLS directly to filebeat but it seems that fortinet filebeat module does not include TLS communication.

Here's a sample of logs by syslog.
Format from syslog is modified as you said.

Apr 12 08:55:33 xxx date=2021-04-12 time=08:50:13 devname="xxx" devid="xxx" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1618210213 srcip=xxx srcport=50006 srcintf="xxx" srcintfrole="lan" dstip=xxx dstport=443 dstintf="port2" dstintfrole="wan" poluuid="dbf92dee-8d7e-51e7-c7f8-cbe9285a5beb" sessionid=137444923 proto=6 action="close" policyid=71 policytype="policy" service="HTTPS" dstcountry="France" srccountry="Reserved" trandisp="snat" transip=xxx transport=50006 appid=42533 app="Google.Services" appcat="General.Interest" apprisk="elevated" applist="xxx" appact="detected" duration=320 sentbyte=3408 rcvdbyte=8078 sentpkt=27 rcvdpkt=31 utmaction="allow" countweb=1 countapp=2 sentdelta=2615 rcvddelta=4799 devtype="Router/NAT Device" devcategory="None" mastersrcmac="xxx" srcmac="xxx" srcserver=0

Soooo, i understand why i 've got this issue with grok.
Is it possible to communicate forti logs with forti filebeat module with TLS? (seems not )

The project is in fine to have the beautiful graphs from this project

I have read all those scripts and i can choose input syslog for logstash.
Is it possible to communicate forti logs with logstash input syslog with TLS? (seems not again )

Thanks

Marius_Iversen · April 14, 2021, 11:11am

Hello again @Alex_Lum

Ah so that explains it. The TLS and TCP actually uses a totally different way to communicate when it comes from fortinet, support for tcp and tls was added in 7.12 here: [Filebeat] rfc6587 framing for fortinet firewall by leehinman · Pull Request #23837 · elastic/beats · GitHub

So if you use the 7.12 version, and set the input type to tcp and configure the TLS settings it should work just fine

Something like this in the fortinet.yml under the firewall module, together with the rest of your settings, though remove things like client auth if its not used:

ssl:
  enabled: true
  certificate: "foo.crt"
  key: "foo.key"
  client_authentication: true
  verification_mode: true # or none if it shouldn't check the hostname of the cert

Alex_Lum · April 14, 2021, 11:58am

Thanks @Marius_Iversen ,
I ' am updating 7.12 so i will test that.
Thanks

Alex_Lum · April 14, 2021, 1:26pm

I have updated filebeat and made configuration but in logs :

 Exiting: Failed to start crawler: creating module reloader failed: could not create module registry for filesets: fileset fortinet/ssl is configured but doesn't exist

Documentation seems not to be updated for fortinet filebeat module

Marius_Iversen · April 14, 2021, 3:03pm

I think at that point it might just be down to a malformed fortinet.yml (the one changed from ./modules.d folder).

Would you be able to provide a copy of it? Formatted as a codeblock so we can see the indentations etc as well.

Alex_Lum · April 14, 2021, 3:23pm

OK @Marius_Iversen , here's my fortinet.yml on modules.d

# Module: fortinet
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-fortinet.html

- module: fortinet
  firewall:
    enabled: true

    # Set which input to use between tcp, udp (default) or file.
    var.input: tcp

    # The interface to listen to syslog traffic. Defaults to
    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
    var.syslog_host: 0.0.0.0

    # The port to listen for syslog traffic. Defaults to 9004.
    var.syslog_port: 9004

  ssl:
    enabled: true
    certificate: "/etc/filebeat/xxx.pem"
    key: "/etc/filebeat/xxx.key"
    client_authentication: true
    verification_mode: none # or none if it shouldn't check the hostname of the cert

  clientendpoint:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    # var.input: udp
    # var.syslog_host: localhost
    # var.syslog_port: 9510

    # Set paths for the log files when file input is used.
    # var.paths:

    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: local

  fortimail:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    # var.input: udp
    # var.syslog_host: localhost
    # var.syslog_port: 9529

    # Set paths for the log files when file input is used.
    # var.paths:

    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: local

  fortimanager:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    # var.input: udp
    # var.syslog_host: localhost
    # var.syslog_port: 9530

    # Set paths for the log files when file input is used.
    # var.paths:

    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: local

Marius_Iversen · April 14, 2021, 3:56pm

Alex_Lum:

# Module: fortinet
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-fortinet.html

- module: fortinet
  firewall:
    enabled: true

    # Set which input to use between tcp, udp (default) or file.
    var.input: tcp

    # The interface to listen to syslog traffic. Defaults to
    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
    var.syslog_host: 0.0.0.0

    # The port to listen for syslog traffic. Defaults to 9004.
    var.syslog_port: 9004

  ssl:
    enabled: true
    certificate: "/etc/filebeat/xxx.pem"
    key: "/etc/filebeat/xxx.key"
    client_authentication: true
    verification_mode: none # or none if it shouldn't check the hostname of the cert

Could you try:

# Module: fortinet
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-fortinet.html

- module: fortinet
  firewall:
    enabled: true

    # Set which input to use between tcp, udp (default) or file.
    var.input: tcp

    # The interface to listen to syslog traffic. Defaults to
    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
    var.syslog_host: 0.0.0.0

    # The port to listen for syslog traffic. Defaults to 9004.
    var.syslog_port: 9004

    var.ssl:
      enabled: true
      certificate: "/etc/filebeat/xxx.pem"
      key: "/etc/filebeat/xxx.key"
      client_authentication: true
      verification_mode: none # or none if it shouldn't check the hostname of the cert

I added both the var.ssl and changed the indentation so that its under the firewall line, instead of on the same indentation

Alex_Lum · April 15, 2021, 8:12am

Ok @Marius_Iversen .

Tested, chain syslog > filebeat > ES > kib OK.

But messages are still encrypted :

Provided Grok expressions do not match field value: [\u0016\u0003\u0001\u0001�\u0001\u0000\u0001�\u0003\u0003��\u0004�#\u0006�^\u0006Jn�M�j\u001A\u0002M\r��\t���ҏ���)C \u0000{\u001F�h7\r���9�-H;���]

Is there an option to add CA certificate like logstash or winlogbeat?

system · May 13, 2021, 8:13am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Provided expression do not match field value filebeat 8.15.0 fortinet module Beats beats-module , filebeat	1	24	August 29, 2024
Provided Grok expressions do not match field value: Fortinet 7.12.1 Beats filebeat	9	3095	May 27, 2021
[CheckPoint] [Filebeat 7.8] Provided Grok expressions do not match field value Beats filebeat	1	505	July 21, 2020
Provided Grok expressions do not match field value: Beats filebeat	1	402	February 4, 2021
NGINX logs - Provided Grok expressions do not match field value Beats filebeat	7	4991	May 16, 2018

FILEBEAT module Fortinet : Provided Grok expressions do not match field value

Related Topics