Filebeat Fortinet Module + Kibana SIEM

Elastic Stack Beats
1

Hello,

I'm using the Fortinet module with Filebeat on Linux, logs are flowing and making their way into Elastic. Is there something I need to do to have these show in SIEM? The Fortinet module docs https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-fortinet.html say the fields are mapped to ECS, so these logs should automatically show in Kibana SIEM? I don't see anything. I've checked siem:defaultIndex in Kibana and that lists filebeat-*. Any ideas on how I can get my Fortinet data to show up in SIEM?

Regards

2

Hello @tjfred

Thanks for testing out the fortinet module!

It does sound quite weird that it would not show up in the SIEM application, most likely there must be a field missing that the SIEM app expects.

Would you be able to share a single document that originated from my module? Feel free to anonymise any identifiable values if you want. To see which fields are populated and such.
The easiest way would be use use the Discover page, and filter on module name.

3

@Marius_Iversen thanks for your response, please find document below:

{
  "_index": "filebeat-7.7.1-2020.06.10-000001",
  "_type": "_doc",
  "_id": "bkL7nXIB5pEhvMFhWMuB",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "hostname": "xx",
      "id": "89ad49b4-bcd9-41c1-b4fa-2d004c365ec5",
      "type": "filebeat",
      "ephemeral_id": "1cb6648a-ed5d-4865-8b7c-13b91deac958",
      "version": "7.7.1"
    },
    "_temp": {
      "time": "2020-06-10 12:26:31 UTC+1:00"
    },
    "log": {
      "source": {
        "address": "xx"
      }
    },
    "syslog5424_sd": "timestamp=1591788391 tz=\"UTC+1:00\" devname=\"xx\" devid=\"xx\" vd=\"root\" date=2020-06-10 time=12:26:31 logid=\"0000000013\" type=\"traffic\" subtype=\"forward\" level=\"notice\" eventtime=1591788391 srcip=xx srcname=\"ubuntu\" srcport=xx srcintf=\"port1\" srcintfrole=\"lan\" dstip=xx dstport=xx dstintf=\"xx\" dstintfrole=\"undefined\" poluuid=\"xx\" sessionid=3971007 proto=17 action=\"accept\" policyid=103 policytype=\"policy\" service=\"DNS\" dstcountry=\"Reserved\" srccountry=\"Reserved\" trandisp=\"noop\" duration=180 sentbyte=88 rcvdbyte=104 sentpkt=1 rcvdpkt=1 vpn=\"xx\" vpntype=\"ipsec-static\" appcat=\"unscanned\" devtype=\"Linux PC\" devcategory=\"None\" osname=\"Linux\" osversion=\"Debian\" mastersrcmac=\"xx\" srcmac=\"xx\" srcserver=1",
    "message": "<189>timestamp=1591788391 tz=\"UTC+1:00\" devname=\"xx\" devid=\"xx\" vd=\"root\" date=2020-06-10 time=12:26:31 logid=\"0000000013\" type=\"traffic\" subtype=\"forward\" level=\"notice\" eventtime=1591788391 srcip=xx srcname=\"ubuntu\" srcport=xx srcintf=\"port1\" srcintfrole=\"lan\" dstip=xx dstport=xx dstintf=\"xx\" dstintfrole=\"undefined\" poluuid=\"xx\" sessionid=3971007 proto=17 action=\"accept\" policyid=103 policytype=\"policy\" service=\"DNS\" dstcountry=\"Reserved\" srccountry=\"Reserved\" trandisp=\"noop\" duration=180 sentbyte=88 rcvdbyte=104 sentpkt=1 rcvdpkt=1 vpn=\"xx\" vpntype=\"ipsec-static\" appcat=\"unscanned\" devtype=\"Linux PC\" devcategory=\"None\" osname=\"Linux\" osversion=\"Debian\" mastersrcmac=\"xx\" srcmac=\"xx\" srcserver=1\n",
    "fileset": {
      "name": "firewall"
    },
    "error": {
      "message": "Invalid ID for ZoneOffset, non numeric characters found: +1:00"
    },
    "tags": [
      "fortinet-firewall"
    ],
    "input": {
      "type": "udp"
    },
    "observer": {
      "product": "Fortigate",
      "vendor": "Fortinet",
      "type": "firewall"
    },
    "@timestamp": "2020-06-10T11:26:33.580Z",
    "ecs": {
      "version": "1.5.0"
    },
    "service": {
      "type": "fortinet"
    },
    "syslog5424_pri": "189",
    "fortinet": {
      "firewall": {
        "devid": "xx",
        "date": "2020-06-10",
        "srcip": "xx",
        "srcintfrole": "lan",
        "poluuid": "xx",
        "dstport": "xx",
        "tz": "UTC+1:00",
        "eventtime": "1591788391",
        "sessionid": "3971007",
        "sentpkt": "1",
        "type": "traffic",
        "devcategory": "None",
        "srccountry": "Reserved",
        "duration": "180",
        "policyid": "103",
        "subtype": "forward",
        "mastersrcmac": "xx",
        "action": "accept",
        "srcmac": "xx",
        "devname": "xx",
        "dstip": "xx",
        "dstintf": "xx",
        "trandisp": "noop",
        "osname": "Linux",
        "timestamp": "1591788391",
        "srcintf": "port1",
        "sentbyte": "88",
        "level": "notice",
        "policytype": "policy",
        "srcserver": "1",
        "vpntype": "ipsec-static",
        "vd": "root",
        "dstintfrole": "undefined",
        "appcat": "unscanned",
        "devtype": "Linux PC",
        "srcname": "ubuntu",
        "vpn": "xx",
        "service": "DNS",
        "proto": "17",
        "srcport": "xx",
        "logid": "0000000013",
        "time": "12:26:31",
        "dstcountry": "Reserved",
        "osversion": "Debian",
        "rcvdbyte": "104",
        "rcvdpkt": "1"
      }
    },
    "host": {
      "hostname": "xx",
      "os": {
        "kernel": "4.19.0-9-amd64",
        "codename": "buster",
        "name": "Debian GNU/Linux",
        "family": "debian",
        "version": "10 (buster)",
        "platform": "debian"
      },
      "containerized": false,
      "ip": [
        "xx",
        "xx"
      ],
      "name": "xx",
      "id": "xx",
      "mac": [
        "xx"
      ],
      "architecture": "x86_64"
    },
    "event": {
      "timezone": "UTC+1:00",
      "module": "fortinet",
      "dataset": "fortinet.firewall"
    }
  },
}

I've removed some things and replaced with xx, you can assume the original data there was correct. This throws another question, is there some reason as to why the original message is stored twice in very similar format? can i remove one of these somehow as I imagine when I ramp things up it will cost space.

Thanks for your assistance with this!

4

I think we have 2 issues here:

  1. I have been able to reproduce the lack of content in SIEM when ingesting fortinet data that I am currently looking into.
  2. The duplicate messages are supposed to be deleted, but you can see that you have a parsing issue, since we also have a error message here, so most likely it was not finished parsing and left multiple fields there.

Here is an example of the output you should expect, without the different agent fields:

    "@timestamp": "2020-04-23T12:17:48.000-05:00",
    "destination.as.number": 15169,
    "destination.as.organization.name": "Google LLC",
    "destination.bytes": 1130,
    "destination.geo.continent_name": "North America",
    "destination.geo.country_iso_code": "US",
    "destination.geo.location.lat": 37.751,
    "destination.geo.location.lon": -97.822,
    "destination.ip": "8.8.8.8",
    "destination.port": 443,
    "event.action": "ftgd_blk",
    "event.category": [
        "network"
    ],
    "event.code": "0316013056",
    "event.dataset": "fortinet.firewall",
    "event.kind": "event",
    "event.module": "fortinet",
    "event.outcome": "success",
    "event.start": "2020-04-18T12:17:49.052-05:00",
    "event.timezone": "-0500",
    "event.type": [
        "denied"
    ],
    "fileset.name": "firewall",
    "fortinet.firewall.action": "blocked",
    "fortinet.firewall.authserver": "elasticauth",
    "fortinet.firewall.cat": "76",
    "fortinet.firewall.dstintfrole": "wan",
    "fortinet.firewall.method": "domain",
    "fortinet.firewall.reqtype": "direct",
    "fortinet.firewall.sessionid": "1234",
    "fortinet.firewall.srcintfrole": "lan",
    "fortinet.firewall.subtype": "webfilter",
    "fortinet.firewall.type": "utm",
    "fortinet.firewall.vd": "root",
    "input.type": "log",
    "log.level": "warning",
    "log.offset": 0,
    "message": "URL belongs to a denied category in policy",
    "network.bytes": 2282,
    "network.direction": "outgoing",
    "network.iana_number": "6",
    "network.protocol": "https",
    "observer.egress.interface.name": "wan1",
    "observer.ingress.interface.name": "port1",
    "observer.name": "testswitch1",
    "observer.product": "Fortigate",
    "observer.serial_number": "somerouterid",
    "observer.type": "firewall",
    "observer.vendor": "Fortinet",
    "related.ip": [
        "192.168.2.1",
        "8.8.8.8"
    ],
    "related.user": [
        "elasticuser"
    ],
    "rule.category": "Internet Telephony",
    "rule.id": "100602",
    "rule.ruleset": "elasticruleset",
    "service.type": "fortinet",
    "source.bytes": 1152,
    "source.ip": "192.168.2.1",
    "source.port": 61930,
    "source.user.group.name": "elasticgroup",
    "source.user.name": "elasticuser",
    "tags": [
        "fortinet-firewall"
    ],
    "url.domain": "elastic.co",
    "url.path": "/config/"
},
5

Thanks @Marius_Iversen - the parsing issue is this one maybe? https://github.com/elastic/beats/issues/19010 is there a fix or a workaround i can make? i already made the changes as put forward by user usp-npe

Regards

6

Tested the SIEM part again now, currently it is not picked up for "hosts" because in theory this is not a specific host, but actually marked as a "observer", the host and destination is commonly the sources generating the network traffic. The network tab for example is populated with network data on the SIEM page.

I will be looking into how I can make this better for the SIEM page, and in the meantime I will apply the changes proposed in the github issue. Please follow the github issue for updates!

I will try to add a workaround, but worst case scenario you can modify the pipeline that has been installed on the elasticsearch cluster, if you dont want to wait for my PR to come in.

7

Thanks, I'll adjust the pipeline to correct the TZ issue for now, and keep my eye on Github. Much appreciated!

8

Hello, I'm experiencing the same issue, the message and syslog5424_sd both populate with the same full message, and none of the fortinet.firewall.x fields convert to actual ECS fields where they could (as your example shows).

Running on 7.8.0 and FortisOS 6.2

9

are you sending your logs via fortianalyzer?

10

The fortigate is sending logs directly to the filebeat box. (It also sends to fortianlayzer, but filebeat is not reading from fortianalyzer).

11

As long as the data is sent directly from fortigate, if there is any issues with timestamps and so it should be resolved soon. If you want to test out the fixes I have also added some links here: Filebeat Fortinet Default Ingest Pipeline fails

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.