Fortinet.tmp.*

Elastic Security SIEM
1

Hi,

I'm trying out fortinet filebeat plugin. Running ELK with 7.10.1 and filebeat 7.10.1
However, I'm getting a strange input values into the documents (in the filebeat-* index)
image

This is my config

- module: fortinet
  firewall:
    enabled: true

    # Set which input to use between tcp, udp (default) or file.
    var.input: udp

    # The interface to listen to syslog traffic. Defaults to
    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
    var.syslog_host: 10.60.1.201

    # The port to listen for syslog traffic. Defaults to 9004.
    var.syslog_port: 9004
    input:
      processors:
        - add_fields:
            target: ''
            fields:
              fortinet.firewall.tz: '+800'

The question is why am I getting the "tmp" part of every data line?
And how do I fix it?

I've verified that the pipelines exist
GET /_ingest/pipeline/filebeat-7.10.0-fortinet-firewall-pipeline

This is the pipeline that does the rename from fortinet.tmp to fortinet.firewall.
But for some reason its not running.

Thanks in advance.

Regards,

Michael

2

Moving this post to the SIEM category; that's where developers of the fortinet module hang out.

Shaunak

3

I don't know much about this pipe and beats but does trying out the simulate API help out to see maybe why it's still adding it?

See if that helps sleuth things out a bit?

4

Thanks for moving it to SIEM category.

5

Did help a bit..

For some reason the message is a bit weird.
When I ran

POST /_ingest/pipeline/filebeat-7.10.0-fortinet-firewall-pipeline/_simulate
{
  "docs": [
    {
      "_index": "filebeat-7.10.0-2021.03.08-000004",
      "_id": "id",
      "_source": {
        "event.module": "fortinet"
      }
    },
    {
      "_index": "filebeat-7.10.0-2021.03.08-000004",
      "_id": "id",
      "_source": {
        "event.module": "fortinet"
      }
    }
  ]
}

{
  "docs" : [
    {
      "doc" : {
        "_index" : "filebeat-7.10.0-2021.03.08-000004",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "event.module" : "fortinet",
          "event" : {
            "ingested" : "2021-03-16T02:16:04.052466924Z"
          },
          "error" : {
            "message" : "field [message] not present as part of path [message]"
          }
        },
        "_ingest" : {
          "timestamp" : "2021-03-16T02:16:04.052466924Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "filebeat-7.10.0-2021.03.08-000004",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "event.module" : "fortinet",
          "event" : {
            "ingested" : "2021-03-16T02:16:04.052482104Z"
          },
          "error" : {
            "message" : "field [message] not present as part of path [message]"
          }
        },
        "_ingest" : {
          "timestamp" : "2021-03-16T02:16:04.052482104Z"
        }
      }
    }
  ]
}

So for some reason my message field looks like this.

image
image1528×72 14 KB

Thanks for your help.
Hopefully somebody from the SIEM category can offer some advise..

6

filebeat -e -d "*"
Gives this when the firewall packet comes it.

2021-03-16T10:24:30.180+0800    DEBUG   [processors]    processing/processors.go:203    Publish event: {
  "@timestamp": "2021-03-16T02:24:30.180Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.10.1",
    "truncated": false,
    "pipeline": "filebeat-7.10.1-fortinet-firewall-pipeline"
  },
  "tags": [
    "fortinet-firewall",
    "forwarded"
  ],
  "fileset": {
    "name": "firewall"
  },
  "service": {
    "type": "fortinet"
  },
  "input": {
    "type": "udp"
  },
  "event": {
    "module": "fortinet",
    "dataset": "fortinet.firewall"
  },
  "agent": {
    "type": "filebeat",
    "version": "7.10.1",
    "hostname": "syslog-gather.test.com",
    "ephemeral_id": "a54da9e3-14eb-42b3-a5b0-cfab666709de",
    "id": "02e9efc7-3d89-479e-bdd7-29d626cccda3",
    "name": "syslog-gather.test.com"
  },
  "message": "<189>date=2021-03-16 time=10:24:30 devname=\"FGT60ETK18021220\" devid=\"FGT60ETK18021220\" logid=\"0001000014\" type=\"traffic\" subtype=\"local\" level=\"notice\" vd=\"root\" eventtime=1615861470 srcip=192.168.123.12 srcport=59594 srcintf=\"internal\" srcintfrole=\"lan\" dstip=192.168.123.255 dstport=10000 dstintf=unknown-0 dstintfrole=\"undefined\" sessionid=70500284 proto=17 action=\"deny\" policyid=0 policytype=\"local-in-policy\" service=\"udp/10000\" dstcountry=\"Reserved\" srccountry=\"Reserved\" trandisp=\"noop\" app=\"udp/10000\" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat=\"unscanned\"",
  "log": {
    "source": {
      "address": "192.168.123.137:4392"
    }
  },
  "ecs": {
    "version": "1.6.0"
  },
  "fortinet": {
    "firewall": {
      "tz": "+800"
    }
  }
}

The message field looks correct to me.

7

The pipelines has this:

        "kv" : {
          "ignore_failure" : false,
          "trim_value" : "\"",
          "field" : "syslog5424_sd",
          "field_split" : """ (?=[a-z\_\-]+=)""",
          "value_split" : "=",
          "prefix" : "fortinet.tmp.",
          "ignore_missing" : true
        }
      },
      {
        "remove" : {
          "if" : "ctx.fortinet?.tmp?.assignip == 'N/A'",
          "ignore_missing" : true,
          "field" : "fortinet.tmp.assignip"
        }
      },
      {
        "rename" : {
          "field" : "fortinet.tmp",
          "target_field" : "fortinet.firewall",
          "ignore_missing" : true
        }
      },

It does the KV part.. as I see the fields split into
fortinet.tmp.
But it skips the rename part where it renames fortinet.tmp to fortinet.firewall.

Please help.

Regards,

Michael

8

So I've found out that without the processor lines it works.. but with the processor lines it messes up.

    input:
      processors:
        - add_fields:
            target: ''
            fields:
              fortinet.firewall.tz: '+800'
9

Solution or not a solution..
The problem with the solution of

    input:
      processors:
        - add_fields:
            target: ''
            fields:
              fortinet.firewall.tz: 'Asia/Singapore'

Does fix the timestamp correctly.. but when it hits this section of the pipeline

      {
        "rename" : {
          "target_field" : "fortinet.firewall",
          "ignore_missing" : true,
          "field" : "fortinet.tmp"
        }
      },

The rename fails because fortinet.firewall.tz already exist. This causes the rest of the pipeline to fail.
To fix that you need to use a different field name as shown below.

The only way to get all of it to work is to add a new "set" in the pipeline and a custom settings in the config

So the pipeline filebeat-7.10.1-fortinet-firewall-pipeline add first set of fortinet.firewall.tz before the set for event.timezone.

      {
        "set" : {
          "field" : "fortinet.firewall.tz",
          "value" : "{{firewall.tz}}",
          "if" : "ctx.firewall.tz != null"
        }
      },
      {
        "set" : {
          "field" : "event.timezone",
          "value" : "{{fortinet.firewall.tz}}",
          "ignore_empty_value" : true
        }
      },

Then in the fortinet.yml add

    input:
      processors:
        - add_fields:
            target: ''
            fields:
              firewall.tz: "Asia/Singapore"

Hence, you need to add it for every version of filebeat pipeline for the firewall-pipeline
Not ideal.. meaning you need to do for every filebeat version
filebeat setup --pipeline module fortinet
Then edit the pipeline as show and your fortinet.yml before actually ingesting data.

If you want you can also remove the added field by modifying the remove section

        "remove" : {
          "field" : [
            "_temp",
            "message",
            "syslog5424_sd",
            "syslog5424_pri",
            "fortinet.firewall.tz",
            "fortinet.firewall.date",
            "fortinet.firewall.eventtime",
            "fortinet.firewall.time",
            "fortinet.firewall.duration",
            "host",
            "firewall.tz"
          ],
          "ignore_missing" : true
        }
      },

Regards,

Michael

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.