Filebeat using Fortinet module TZ issue

[Derick_Jansen]

Hi,

We are using Filebeat with the Fortinet module to ingest our FW logs via Syslog.
Our Fortigates do not send timezone information in log entries.

The logs all have this format

<189>date=2020-10-09 time=14:06:56 devname=\"****-fw1\"

Additionally, the FW is not set to GMT. The result is that all our events are logged 13 hours forward (We are at GMT+13)

Any way to explicitly set the timezone?

Thanks and regards!

[Ameer_Mukadam]

Glad I am not the only one I am also having the same issue. Our fortigates are set to use +5:30 and all my logs are +5:30 hr in the future.

[Derick_Jansen]

Any help? If not setting the TZ explicitly, perhaps an ingest change?

[Derick_Jansen]

I tried both File and Syslog ingestion. Problem exists in both cases. Upgraded to 7.9.3 but no difference.

[Iker (Iker Saint)]

If you are ingesting the logs through Logstash set the correct timezone pretty easy

    date {
        	match => ["@timestamp", "UNIX"]
        	timezone => "Pacific/Tongatapu"
     	        target => "@timestamp"
     }

If you are ingesting directly to ES, there are several options, using the option var.tz_offset for the module, overwrite the event.timezone field with a processor or use a pipeline

Fortinet module | Filebeat Reference [7.9] | Elastic

Be aware that in some versions (logver=60) Fortinet includes the timezone.

[Derick_Jansen]

Hi @Iker

Thanks for the reply. We are ingesting directly but I will test via Logstash.

@Iker

var.tz_offset

I did test var.tz_offset and can confirm that it is only implemented for the clientendpoint fileset and not the firewall fileset.

[BenB196]

Looking at the pipeline 'filebeat-7.9.1-fortinet-firewall-pipeline' it looks like the timezone is read from the field: 'fortinet.firewall.tz'. You could try setting this field within Filebeat processor. This is an example value for what fortinet.firewall.tz is -0500.

Note: It appears that fortinet.firewall.tz is copied to event.timezone and then dropped within the pipeline, so if you're looking for the field value, look at event.timezone

[Derick_Jansen]

Ok here are my findings

event.start - Nov 5, 2020 @ 19:20:00.000 - Correct
event.timezone - +13:00 - Correct

@timestamp - Nov 6, 2020 @ 08:20:00.000 - Incorrect

In short, the timestamp is incorrect

[BenB196]

According to the below part of the firewall pipeline, it should've set the @timestamp field to be correct.

{
"date": {
  "if": "ctx.fortinet?.firewall?.tz != null",
  "field": "_temp.time",
  "target_field": "@timestamp",
  "formats": [
    "yyyy-MM-dd HH:mm:ss",
    "yyyy-MM-dd HH:mm:ss Z",
    "yyyy-MM-dd HH:mm:ss z",
    "ISO8601"
  ],
  "timezone": "{{fortinet.firewall.tz}}"
}

Since the below is fairly similar to setting the event.start field.

{
"date": {
  "target_field": "event.start",
  "formats": [
    "UNIX"
  ],
  "timezone": "{{fortinet.firewall.tz}}",
  "if": "ctx.fortinet?.firewall?.tz != null && (ctx.fortinet?.firewall?.eventtime).length() <= 11",
  "field": "fortinet.firewall.eventtime"
}

What you can do if setting the fortinet.firewall.tz doesn't work, you can at the very end of the firewall pipeline, copy event.start field to @timestamp field. (Note: This might cause some issues as according to ECS they technically are two different values, but its a potential work around)

[Derick_Jansen]

@Iker @BenB196 @Ameer_Mukadam

Thanks a lot for all the help. I have now solved this issue.

fortinet.yml

- module: fortinet
  firewall:
    input:
      processors:
        - add_fields:
            target: ''
            fields:
              fortinet.firewall.tz: '+1300'
    enabled: true
    var.input: udp
    var.syslog_port: 5514

I also tried add_locale processor as below.

processors:
  - add_locale: ~

This would be preferable as daylight saving changes the offset but add_locale uses "+13:00" as the format which the pipeline does not support.

I might use the script processor to modify the add_local value.

[BenB196]

Another thing to potentially note. You might want to try setting the Canonical ID for the Timezone. Even though my original example was the standard offset, I've realized that the Canonical ID should also work for the fortinet.firewall.tz. This is because the pipeline is what is using the timezone, and the date processor should accept the Canonical ID as well as the standard offset, which should help overcome your DST issue.

[Ameer_Mukadam]

@Derick_Jansen

Nov 6, 2020 @ 08:20:00.000

Thank you very very much this was so frustrating and even the support couldn't help me in this, you fixed it.