Filebeat: Cannot parse certain FTD messages, bug?

SjoerdFurth · April 7, 2020, 5:50pm

At work we wanted to monitor our Cisco FTD using Elasticsearch and try the SIEM option.
After setting this up with version 7.5.2 and 7.6.1 (when it released), I noticed some of the syslog messages not to be parsed correctly.

One of the reasons is already fixed in GH issue 16889 due to some of the fields missing. The other issues I will demonstrate below.

The syslog messages that are not working are:

Mar 13 15:47:15 CISCOFTD %FTD-4-106023: Deny icmp src INSIDE:172.31.98.44 dst OUTSIDE:100.66.124.24 (type 8, code 0) by access-group "CSM_FW_ACL_" [0x900503bb, 0xf8ee23a1]

The message above produces the error:

Unable to find match for dissect pattern: %{event.outcome} %{network.transport} src %{_temp_.cisco.source_interface}:%{source.address}/%{source.port} dst %{_temp_.cisco.destination_interface}:%{destination.address}/%{destination.port} %{} access%{}group \\\"%{_temp_.cisco.list_id}\\\"%{} against source: Deny icmp src INSIDE:172.31.98.44 dst OUTSIDE:100.66.124.24 (type 8, code 0) by access-group \\\"CSM_FW_ACL_\\\" [0x900503bb, 0xf8ee23a1]

This is due to Grok patterns not knowing about the type 8, code 0 part

Mar 13 02:59:17 CISCOFTD %FTD-6-302014: Teardown TCP connection 108430644 for INSIDE:172.31.98.44/51568 to OUTSIDE:100.66.124.24/443 duration 82:03:50 bytes 1159060 TCP FINs from OUTSIDE

The message above produces the error:

Provided Grok expressions do not match field value: [Teardown TCP connection 108430644 for INSIDE:172.31.98.44/51568 to OUTSIDE:100.66.124.24/443 duration 82:03:50 bytes 1159060 TCP FINs from OUTSIDE]

This is due to the session time being longer then 24 hours. I have seen sessions lasting even longer than 100 hours.

Mar 13 01:25:42 CISCOFTD %FTD-6-302014: Teardown TCP connection 111775548 for INSIDE:172.31.98.44/59058 to OUTSIDE:100.66.124.24/902 duration 0:22:01 bytes 2415600809 TCP FINs from INSIDE

The message above produces the error message:
For input string: \"2415600809\"

Since I had some spare time, I went and fixed the errors myself. I have published the changes here:

github.com

SjoerdFurth/beats/blob/7.6/x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml

description: "Pipeline for Cisco {< .internal_PREFIX >} logs"
processors:
  #
  # Parse the syslog header
  #
  # This populates the host.hostname, process.name, timestamp and other fields
  # from the header and stores the message contents in log.original.
  - grok:
      field: message
      patterns:
        - "(?:%{SYSLOG_HEADER})?\\s*%{GREEDYDATA:log.original}"
      pattern_definitions:
        SYSLOG_HEADER: "(?:%{SYSLOGFACILITY}\\s*)?(?:%{FTD_DATE:_temp_.raw_date}:?\\s+)?(?:%{PROCESS_HOST}|%{HOST_PROCESS})(?:{DATA})?%{SYSLOG_END}?"
        SYSLOGFACILITY: "<%{NONNEGINT:syslog.facility:int}(?:.%{NONNEGINT:syslog.priority:int})?>"
        # Beginning with version 6.3, Firepower Threat Defense provides the option to enable timestamp as per RFC 5424.
        FTD_DATE: "(?:%{TIMESTAMP_ISO8601}|%{ASA_DATE})"
        ASA_DATE: "(?:%{DAY} )?%{MONTH}  *%{MONTHDAY}(?: %{YEAR})? %{TIME}(?: %{TZ})?"
        PROCESS: "(?:[^%\\s:\\[]+)"
        SYSLOG_END: "(?:(:|\\s)\\s+)"
        # exactly match the syntax for firepower management logs

This file has been truncated. show original

Can you confirm this is indeed a bug in the pipelines and check if the changes are according to the guidelines?

system · May 5, 2020, 5:50pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.