Time Zone in CEF FB Module

I am new to all this, so bear with me if i use the wrong terminology.

Working on getting Fortigate logs into ES.

Currently the path the events take are:

Fortgate -> FortiAnalyzer (forwarded in CEF format) -> FileBeat(CEF Module) -> Logstash -> ES

Data is flowing fine, except the time in the events forwarded from the FortiAnalyzer are in EST (UTC -5), and somewhere during ingestion, it gets tagged with a UTC timestamp.

This causes the CEF events in Kibana to look like they happened 5 hours earlier.

Can I edit a pipeline json file somewhere so the conversion to UTC time for the timestamp does not occur, or is there another way of accomplishing the task?

NOTE: Everything here is based on ES v7.5.1 basic license with everything running on a single node.

Update (NOTE the ultimate goal is to get this stuff into ES SIEM):

FortiAnalyzer has the ability to forward logs in three formats

  • FortiAnalyser
  • Syslog
  • Common Event Format (CEF)

PROBLEM
CEF formatted messages:
Are being indexed with an incorrect timestamp. bad
Many of the fields appear to map properly to the ECS and are availabe in the SIEM dashboards. bad

SYSLOG formatted messages:
Are being indexed with a correct timestamp good
None of the fields are being mapped bad

Regardless of format (CEF or syslog), the data received is being mapped properly to the ECS field names. It does appear that more of the CEF formatted data is being recognized though.

NOTE - FortiAnalyzer actually sends three types of logs: "Traffic", "Event" & "UTM". I originally was sending all three types, but am now just sending Traffic logs since i figured that would be easiest to deal with just one first and they data in that type appears the most common.

QUESTION
What would be an easier path to take to get the FortiAnalyzer data into ES so that it can be useful to the SIEM app?

Using CEF formatted logs (fix the timestamp issue and figure out how to ingest the remaining fields so that it maps properly to the ECS)?

or

Using SYSLOG formatted logs (figure out how to ingest the data so that all fields map properly to the ECS).

if someone can point me to instructions on pasting json formatted documents into a discussion thread, I will include samples of how CEF and syslog formatted messages look once they make their way into the filebeat index.

Hello @culprit ,
What does the CEF date look like, can you give an example?

if someone can point me to instructions on pasting json formatted documents into a discussion thread, I will include samples of how CEF and syslog formatted messages look once they make their way into the filebeat index.

on linux you can use jq to make you json prettier before posting if it is what you asked for:

echo '{"fruit": "Apple","size": "Large","color": "Red"}' | jq
or
cat "your file containing json" | jq

copy the output (your json will be formatted), select it then use Preformatted text (</>) on the forum menu bar

{
  "fruit": "Apple",
  "size": "Large",
  "color": "Red"
}

(or use a site like this one https://jsonformatter.curiousconcept.com/ instead of jq to format you json)

Does this help?

{
  "_index": "filebeat-7.5.1-2020.01.14-000001",
  "_type": "_doc",
  "_id": "CnhMrm8BNzoPuM1b3DyI",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "source": {
      "port": 55623,
      "bytes": 6267,
      "nat": {
        "ip": "66.194.18.134",
        "port": 55623
      },
      "ip": "10.212.134.200"
    },
    "fileset": {
      "name": "log"
    },
    "ecs": {
      "version": "1.1.0"
    },
    "observer": {
      "hostname": "FG100D_1",
      "vendor": "Fortinet",
      "version": "5.6.11,build1700 (GA)",
      "product": "FortiGate-100D"
    },
    "destination": {
      "port": 443,
      "bytes": 4601,
      "ip": "142.93.181.170",
      "user": {
        "name": "myname"
      }
    },
    "log": {
      "source": {
        "address": "192.168.11.19:55682"
      }
    },
    "input": {
      "type": "syslog"
    },
    "service": {
      "type": "cef"
    },
    "process": {
      "program": "CEF"
    },
    "tags": [
      "cef",
      "beats_input_codec_plain_applied"
    ],
    "cef": {
      "device": {
        "event_class_id": "0000000020",
        "vendor": "Fortinet",
        "version": "5.6.11,build1700 (GA)",
        "product": "FortiGate-100D"
      },
      "severity": "5",
      "version": "0",
      "name": "forward traffic accept",
      "extensions": {
        "transportProtocol": "6",
        "deviceOutboundInterface": "wan1",
        "destinationAddress": "142.93.181.170",
        "deviceInboundInterface": "ssl.root",
        "deviceHostName": "FG100D_1",
        "deviceSeverity": "notice",
        "bytesIn": "6267",
        "bytesOut": "4601",
        "externalID": "105891068",
        "ad": {
          "policyid": "16",
          "trandisp": "snat",
          "app": "MyApp",
          "sentpkt": "50",
          "appid": "32752",
          "rcvddelta": "215\n\u0000",
          "rcvdpkt": "48",
          "duration": "1355",
          "apprisk": "medium",
          "policytype": "policy",
          "sentdelta": "195",
          "appcat": "Storage.Backup",
          "dstcountry": "United States",
          "vd": "root",
          "applist": "BTM Default App Control",
          "logid": "0000000020",
          "srccountry": "Reserved",
          "poluuid": "5f59b0cc-18d1-51e8-8d89-949f8dbe3ef1",
          "subtype": "forward",
          "eventtime": "1579177272",
          "srcintfrole": "lan",
          "dstintfrole": "wan"
        },
        "sourceTranslatedAddress": "66.194.18.134",
        "deviceEventCategory": "traffic",
        "applicationProtocol": "HTTPS",
        "deviceAction": "accept",
        "sourceTranslatedPort": "55623",
        "logver": "56",
        "destinationPort": "443",
        "deviceCustomString6": "VPN Users",
        "sourcePort": "55623",
        "startTime": "Jan 16 2020 07:21:08",
        "deviceExternalId": "FG100D3G15820636",
        "destinationUserName": "MyName",
        "sourceAddress": "10.212.134.200"
      }
    },
    "syslog": {},
    "message": "forward traffic accept",
    "agent": {
      "hostname": "btm-node1",
      "ephemeral_id": "ab596b42-d39a-491c-ad48-8d8a51ea7d5e",
      "version": "7.5.1",
      "type": "filebeat",
      "id": "eb878b3e-8816-434e-a4be-6406f67509a3"
    },
    "@timestamp": "2020-01-16T07:21:08.000Z",
    "hostname": "FG100D",
    "host": {
      "containerized": false,
      "hostname": "btm-node1",
      "os": {
        "platform": "ubuntu",
        "kernel": "4.15.0-74-generic",
        "family": "debian",
        "version": "18.04.3 LTS (Bionic Beaver)",
        "name": "Ubuntu",
        "codename": "bionic"
      },
      "architecture": "x86_64",
      "name": "btm-node1",
      "id": "dfa0f552294645e6a57be2f18cb61e7f"
    },
    "network": {
      "community_id": "1:vp+V6zhVgNtwsPHYn0vcTTxq1v0=",
      "transport": "6",
      "application": "HTTPS"
    },
    "event": {
      "dataset": "cef.log",
      "original": "CEF:0|Fortinet|FortiGate-100D|5.6.11,build1700 (GA)|0000000020|forward traffic accept|5|start=Jan 16 2020 07:21:08 logver=56 deviceExternalId=FG100D3G15820636 dvchost=FG100D_1 ad.vd=root ad.logid=0000000020 cat=traffic ad.subtype=forward deviceSeverity=notice ad.eventtime=1579177272 src=10.212.134.200 spt=55623 deviceInboundInterface=ssl.root ad.srcintfrole=lan dst=142.93.181.170 dpt=443 deviceOutboundInterface=wan1 ad.dstintfrole=wan ad.poluuid=5f59b0cc-18d1-51e8-8d89-949f8dbe3ef1 externalID=105891068 proto=6 act=accept duser=MyName cs6=VPN Users ad.policyid=16 ad.policytype=policy app=HTTPS ad.dstcountry=United States ad.srccountry=Reserved ad.trandisp=snat sourceTranslatedAddress=66.194.18.134 sourceTranslatedPort=55623 ad.appid=32752 ad.app=MyApp ad.appcat=Storage.Backup ad.apprisk=medium ad.applist=BTM Default App Control ad.duration=1355 out=4601 in=6267 ad.sentpkt=50 ad.rcvdpkt=48 ad.sentdelta=195 ad.rcvddelta=215\n\u0000",
      "code": "0000000020",
      "severity": 5,
      "start": "2020-01-16T07:21:08.000Z",
      "module": "cef",
      "action": "accept"
    }
  },
  "fields": {
    "event.start": [
      "2020-01-16T07:21:08.000Z"
    ],
    "suricata.eve.flow.start": [
      "2020-01-16T07:21:08.000Z"
    ],
    "suricata.eve.timestamp": [
      "2020-01-16T07:21:08.000Z"
    ],
    "@timestamp": [
      "2020-01-16T07:21:08.000Z"
    ]
  },
  "highlight": {
    "event.module": [
      "@kibana-highlighted-field@cef@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1579159268000
  ]
}

Above was an event_class_id (0000000020) that doesnt happen very often. I need those indexed properly but one similar to below (0000000013) occurs probably more than 90% of the time.

{
  "_index": "filebeat-7.5.1-2020.01.14-000001",
  "_type": "_doc",
  "_id": "CXhMrm8BNzoPuM1b3DyI",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "ecs": {
      "version": "1.1.0"
    },
    "fileset": {
      "name": "log"
    },
    "source": {
      "port": 58154,
      "bytes": 5854,
      "nat": {
        "ip": "66.194.18.134",
        "port": 58154
      },
      "ip": "10.201.0.22",
      "domain": "Bowens-iPad"
    },
    "input": {
      "type": "syslog"
    },
    "observer": {
      "hostname": "FG100D_1",
      "vendor": "Fortinet",
      "version": "5.6.11,build1700 (GA)",
      "product": "FortiGate-100D"
    },
    "log": {
      "source": {
        "address": "192.168.11.19:55682"
      }
    },
    "process": {
      "program": "CEF"
    },
    "service": {
      "type": "cef"
    },
    "tags": [
      "cef",
      "beats_input_codec_plain_applied"
    ],
    "destination": {
      "port": 443,
      "bytes": 1622,
      "ip": "198.200.171.198"
    },
    "cef": {
      "device": {
        "event_class_id": "0000000013",
        "vendor": "Fortinet",
        "version": "5.6.11,build1700 (GA)",
        "product": "FortiGate-100D"
      },
      "severity": "5",
      "version": "0",
      "name": "forward traffic client-rst",
      "extensions": {
        "transportProtocol": "6",
        "deviceOutboundInterface": "wan1",
        "destinationAddress": "198.200.171.198",
        "deviceInboundInterface": "BTMPublic-5G",
        "deviceHostName": "FG100D_1",
        "deviceSeverity": "notice",
        "bytesIn": "5854",
        "bytesOut": "1622",
        "externalID": "105910840",
        "ad": {
          "policyid": "28",
          "trandisp": "snat",
          "app": "HTTPS.BROWSER",
          "appid": "40568",
          "sentpkt": "13",
          "srcserver": "0\n\u0000",
          "mastersrcmac": "04:52:f3:0a:58:ea",
          "apsn": "FP221C3X15037300",
          "channel": "153",
          "radioband": "802.11ac",
          "devtype": "iPad",
          "duration": "18",
          "apprisk": "medium",
          "ap": "Transactional",
          "policytype": "policy",
          "appcat": "Web.Client",
          "dstcountry": "United States",
          "osname": "iPad",
          "osversion": "iOS 13.2.3",
          "vd": "root",
          "applist": "default",
          "srcmac": "04:52:f3:0a:58:ea",
          "logid": "0000000013",
          "srccountry": "Reserved",
          "countapp": "1",
          "subtype": "forward",
          "srcintfrole": "lan",
          "dstintfrole": "wan",
          "utmaction": "allow",
          "eventtime": "1579177270",
          "countweb": "1",
          "srcssid": "BTMPublic-5G",
          "poluuid": "3c5b4556-1b13-51e8-1a03-e2eabd4d4878"
        },
        "sourceTranslatedAddress": "66.194.18.134",
        "applicationProtocol": "Opentable",
        "deviceEventCategory": "traffic",
        "deviceAction": "client-rst",
        "sourceTranslatedPort": "58154",
        "sourceHostName": "Bowens-iPad",
        "logver": "56",
        "destinationPort": "443",
        "sourcePort": "58154",
        "startTime": "Jan 16 2020 07:21:08",
        "deviceExternalId": "FG100D3G15820636",
        "sourceAddress": "10.201.0.22"
      }
    },
    "syslog": {},
    "message": "forward traffic client-rst",
    "agent": {
      "ephemeral_id": "ab596b42-d39a-491c-ad48-8d8a51ea7d5e",
      "hostname": "btm-node1",
      "version": "7.5.1",
      "type": "filebeat",
      "id": "eb878b3e-8816-434e-a4be-6406f67509a3"
    },
    "@timestamp": "2020-01-16T07:21:08.000Z",
    "hostname": "FG100D",
    "host": {
      "containerized": false,
      "hostname": "btm-node1",
      "os": {
        "platform": "ubuntu",
        "kernel": "4.15.0-74-generic",
        "family": "debian",
        "version": "18.04.3 LTS (Bionic Beaver)",
        "name": "Ubuntu",
        "codename": "bionic"
      },
      "architecture": "x86_64",
      "name": "btm-node1",
      "id": "dfa0f552294645e6a57be2f18cb61e7f"
    },
    "network": {
      "community_id": "1:AIve9tHTa7FGrB3o5GsKveiGAuY=",
      "transport": "6",
      "application": "Opentable"
    },
    "event": {
      "dataset": "cef.log",
      "original": "CEF:0|Fortinet|FortiGate-100D|5.6.11,build1700 (GA)|0000000013|forward traffic client-rst|5|start=Jan 16 2020 07:21:08 logver=56 deviceExternalId=FG100D3G15820636 dvchost=FG100D_1 ad.vd=root ad.logid=0000000013 cat=traffic ad.subtype=forward deviceSeverity=notice ad.eventtime=1579177270 src=10.201.0.22 shost=Bowens-iPad spt=58154 deviceInboundInterface=BTMPublic-5G ad.srcintfrole=lan ad.srcssid=BTMPublic-5G ad.apsn=FP221C3X15037300 ad.ap=Transactional ad.channel=153 ad.radioband=802.11ac dst=198.200.171.198 dpt=443 deviceOutboundInterface=wan1 ad.dstintfrole=wan ad.poluuid=3c5b4556-1b13-51e8-1a03-e2eabd4d4878 externalID=105910840 proto=6 act=client-rst ad.policyid=28 ad.policytype=policy app=Opentable ad.dstcountry=United States ad.srccountry=Reserved ad.trandisp=snat sourceTranslatedAddress=66.194.18.134 sourceTranslatedPort=58154 ad.appid=40568 ad.app=HTTPS.BROWSER ad.appcat=Web.Client ad.apprisk=medium ad.applist=default ad.duration=18 out=1622 in=5854 ad.sentpkt=13 ad.utmaction=allow ad.countweb=1 ad.countapp=1 ad.devtype=iPad ad.osname=iPad ad.osversion=iOS 13.2.3 ad.mastersrcmac=04:52:f3:0a:58:ea ad.srcmac=04:52:f3:0a:58:ea ad.srcserver=0\n\u0000",
      "code": "0000000013",
      "severity": 5,
      "start": "2020-01-16T07:21:08.000Z",
      "module": "cef",
      "action": "client-rst"
    }
  },
  "fields": {
    "event.start": [
      "2020-01-16T07:21:08.000Z"
    ],
    "suricata.eve.flow.start": [
      "2020-01-16T07:21:08.000Z"
    ],
    "suricata.eve.timestamp": [
      "2020-01-16T07:21:08.000Z"
    ],
    "@timestamp": [
      "2020-01-16T07:21:08.000Z"
    ]
  },
  "highlight": {
    "event.module": [
      "@kibana-highlighted-field@cef@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1579159268000
  ]
}

so it should be Thursday, January 16, 2020 12:21:12 PM? (eventtime in CEF is 1579177272)

maybe you have this problem
sorry, I can't help much, I just happen to be someone with another CEF time issue and I was curious to see if we had the same problem. It doesn't seem so.

The event occurred at 7:21:08 EST. It was indexed as UTC time with no conversion from EST to UTC.

When Kibana displays the document, it takes the UTC time in the index, sees I am EST so it shows the even as occurring at 2:21:08 EST.

Fortigate sends:

CEF:0|Fortinet|FortiGate-100D|5.6.11,build1700 (GA)|0000000013|forward traffic client-rst|5|start=Jan 16 2020 07:21:08 logver=56 (...)

I think the problem (or at least some part of it) is that there is no timezone indication so I guess UTC is assumed

maybe you can try to explicitely set the timezone in filebeat.yml

processors:
  - add_fields:
    target: event
    fields:
      timezone: 'America/New_York'

once you do that and if it works. your kibana will still do conversion. in order to stop that you have to setup timezone on kibana from browser to "America/New_york" and then your bar chart with datehistogram will be off.

I have other modules enabled in filebeat. Those events are being indexed fine. Wouldn't adding

processors:
  - add_fields:
    target: event
    fields:
      timezone: 'America/New_York'

cause all events currently in UTC to now be incorrect?

I will give it a go and see what happens, but looking elsewhere found e reference to a timestamp processor. Could I add that to the the CEF config module and include the timezone option to subtitute in only during CEF processing?

https://www.elastic.co/guide/en/beats/filebeat/7.5/processor-timestamp.html

something like:

processors:
- timestamp:
    field: start_time
    layouts:
      - '2006-01-02T15:04:05Z'
      - '2006-01-02T15:04:05.999Z'
    test:
      - '2019-06-22T16:33:51Z'
      - '2019-11-18T04:59:51.123Z'
    timezone:  "America/New_York"
- drop_fields:
    fields: [start_time]

you could try to set a condition so the processor would only apply to CEF

processors:
  - rename:
      fields:
        - {from: "message", to: "event.original"}
  - decode_cef:
      field: event.original
  - add_fields:
      when:
        has_fields: ['cef']
      target: event
      fields:
        timezone: 'America/New_York'

or (if you plan to receive CEF from other appliances which don't have a timezone problem)

 - equals:
       cef.vendor: "Fortinet"

another option is to rely on the cef.extensions.ad.eventtime but you have to be sure it is always present in your Fortinet logs

Yeah... things I tried so far haven't worked, but I am only a week into learning ES, so still trying to figure out how things are put together. So many different processes involved and associated config files, I am not sure which file I need to be editing to affect certain changes.

From what I can tell so far, other Beat modules have TZ problems. It was somewhat fixed in some, but not all of them. From the docs on the CEF module...

This is a module for receiving Common Event Format (CEF) data over Syslog. When messages are received over the syslog protocol the syslog input will parse the header and set the timestamp value. Then the decode_cef processor is applied to parse the CEF encoded data. The decoded data is written into a cef object field. Lastly any Elastic Common Schema (ECS) fields that can be populated with the CEF data are populated.

...I read somewhere here maybe that TZ issues similar to mine were fixed in syslog but it might be only through the system module.

Since CEF utilizes the syslog input to parse the header and timestamp I was thinking I could do something with the event.timezone field somewhere, but I am not sure what (add/drop field) or where (CEF or Syslog, manifest, config, pipeline, etc.).

I do see the event.timezone field in the beats/filebeat/module/system/syslog/ingest/pipeline.json file. Is this what is used by the CEF module to set the timestamp value? If so what CEF file could I use to set the value for event.timezone?

Giving up for now. Adding processor below was inserting event.timezone field into the record, but timestamp field still wasn't changing.

  processors:
  - rename:
      fields:
        - {from: "message", to: "event.original"}
  - decode_cef:
      field: event.original
  - add_fields:
      when:
        has_fields: ['cef']
      target: event
      fields:
        timezone: 'America/New_York'

Even went so far as adding code I copied from /usr/share/filebeat/module/panw/panos/ingest/pipeline.yml into the CEF ingest pipeline, but that had no effect either. (yeah - i did delete/add the pipeline)

# keep message as log.original.
  - rename:
      field: message
      target_field: log.original

# Set @timestamp to the time when the entry was generated at the data plane.
  - date:
      if: "ctx.event.timezone == null"
      field: "_temp_.generated_time"
      formats:
        - "yyyy/MM/dd HH:mm:ss"
      on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
  - date:
      if: "ctx.event.timezone != null"
      field: "_temp_.generated_time"
      formats:
        - "yyyy/MM/dd HH:mm:ss"
      timezone: "{{ event.timezone }}"
      on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]

# event.created is the time the event was received at the management plane.
  - date:
      if: "ctx.event.timezone == null && ctx.event.created != null "
      field: "event.created"
      target_field: "event.created"
      formats:
        - "yyyy/MM/dd HH:mm:ss"
      on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
  - date:
      if: "ctx.event.timezone != null && ctx.event.created != null "
      field: "event.created"
      target_field: "event.created"
      formats:
        - "yyyy/MM/dd HH:mm:ss"
      timezone: "{{ event.timezone }}"
      on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]

# event.start (traffic only) is the time the session started.
  - date:
     if: "ctx.event.timezone == null && ctx.event.start != null"
     field: "event.start"
     target_field: "event.start"
     formats:
       - "yyyy/MM/dd HH:mm:ss"
     on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
  - date:
      if: "ctx.event.timezone != null && ctx.event.start != null"
      field: "event.start"
      target_field: "event.start"
      timezone: "{{ event.timezone }}"
      formats:
        - "yyyy/MM/dd HH:mm:ss"
      on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]

not optimal but set the firewall to start recording events in GMT for now just to see how things tie together in SIEM.

FWIW... I was poking around git and saw someone was working on a fortigate module to work on the syslog data so may need to wait a bit to see how that turns out. Since I am currently shipping all logs to the fortinanalyzer first, and that could output in CEF, it had some fields already mapping to ECS & SIEM I figured I would go that route. Oh well.

@culprit ok, sorry to hear everything didn't work for you, i'm new at ELK myself so I can't really help a lot
maybe I had the order wrong (I mean if the UTC assumption /convertion is done right at the beginning maybe adding the timezone after CEF decoding doesn't change anything). Someone with more skills and experience at ELK could certainly help. Anyone?