Filebeat Fortinet Default Ingest Pipeline fails

Mercwri · June 26, 2020, 5:41pm

I have multiple Fortinet devices pushing logs to a collector running Filebeat 7.8.0 with the Fortinet module enabled. All of the logs are being ingested but the pipeline fails at decoding/normalizing the timestamps.

I get the following error:

Text '2020-06-26 13:39:00Z' could not be parsed, unparsed text found at index 19

It appears to be an issue with how the _temp.time field is processed by the Ingest Pipeline but I have not to been able to isolate the issue.

Derick_Jansen · June 30, 2020, 10:45pm

I have the same issue. What version of the FortiOS firmware are you running?

tdanno · July 6, 2020, 12:47pm

I'm having the same issue and error message with the time stamp normalizing, on FortiOS 6.2 .

Also, the fields are coming through as all 'fortinet.firewall.' instead of the proper ECS equivalent. i.e., instead of 'source.ip' I'm getting 'fortinet.firewall.srcip' .

Marius_Iversen · July 14, 2020, 1:47pm

@Mercwri @Derick_Jansen @tdanno

Could anyone provide a log sample? I have merged some changes for timestamp parsing that should be included in the next release.

Ref: https://github.com/elastic/beats/issues/19154
https://github.com/elastic/beats/issues/19010
https://github.com/elastic/beats/issues/18707

If you want to test it, you can modify the pipeline.yml or the pipeline on elasticsearch with the changes here:

github.com

elastic/beats/blob/master/x-pack/filebeat/module/fortinet/firewall/ingest/pipeline.yml

description: Pipeline for parsing fortinet firewall logs
processors:
- grok:
    field: message
    patterns:
    - '%{SYSLOG5424PRI}%{GREEDYDATA:syslog5424_sd}$'
- kv:
    field: syslog5424_sd
    field_split: " (?=[a-z\\_\\-]+=)"
    value_split: "="
    prefix: "fortinet.firewall."
    ignore_missing: true
    ignore_failure: false
    trim_value: "\""
- set:
    field: observer.vendor
    value: Fortinet
- set:
    field: observer.product
    value: Fortigate

This file has been truncated. show original

If you look at the date processors here they might have added a third format or something similar to support this.

Mercwri · July 14, 2020, 2:12pm

The revised pipeline appears to have resolved the timestamp processing.

Marius_Iversen · July 14, 2020, 2:28pm

Thanks for confirming!

system · August 11, 2020, 4:29pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.