Zeek Import - Timestamp & Geo Failing

[0x00]

I am attempting to use Filebeat on Ubuntu Linux to import a JSON formatted conn.log as the obfuscated sample is shown below:

{"_path":"conn","_system_name":"sensor","_write_ts":"2019-07-02T15:53:03.511364Z","ts":"2019-07-02T15:52:46.377889Z","uid":"CByC0qkmPzrL8w4Akj","id.orig_h":"12.34.56.78","id.orig_p":64069,"id.resp_h":"98.76.54.32","id.resp_p":135,"proto":"tcp","service":"dce_rpc","duration":12.133461,"orig_bytes":2355,"resp_bytes":395,"conn_state":"SF","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ShADadFf","orig_pkts":9,"orig_ip_bytes":2727,"resp_pkts":7,"resp_ip_bytes":687,"tunnel_parents":[],"orig_l2_addr":"00:11:22:33:44:55","resp_l2_addr":"aa:bb:cc:dd:ee:ff"}

I have the zeek module enabled:

And it successfully imports the log.

However it does "NOT" allow me to use the proper, original, timestamp and instead uses the time listed when the log was processed.

As an additional challenge, it doesn't seem to be adding geo location.

How can I go about getting it to use the right (original conn log time) timestamp and get it to process geo location data against the conn log?

Can I update fields.yml or connection.yml or something of this sort?

[0x00]

I tried adding this to /var/usr/filebeat/module/zeek/connection/ingest/pipeline.json which did not help.

{
  "rename": {
    "field": "@timestamp",
    "target_field": "event.created"
  }
},
{
  "date": {
    "field": "zeek.notice.ts",
    "target_field": "@timestamp",
    "formats": ["yyyy-MM-dd HH:mm:ss"]
  }
},

I tried to add this to /etc/filebeat/fields.yml which also did not help.

- name: connection.ts
  type: date
  description: >
    my timestamp field

And I tried adding this to /var/usr/filebeat/module/zeek/connection/config/connection.yml which clearly also failed.

    - from: "zeek.connection.ts"
      to: "event.created"

Please forgive my naivety around how this is all set up. I am still in the process of learning all of this and trying to figure out how things still go together.

[0x00]

I am still fighting with this but have an update.

Timestamp Resolution (sort of):
I switched from having filebeat send the data straight to elasticsearch and instead started sending it to logstash. This action helped resolve the timestamp problem and I can now select it though I still don't understand why I couldn't do that when sending the data JSON -> Filebeat -> Elasticsearch.

However, geo / geo point / geoip is still not working and I'm fighting with how to get it functioning properly.

1. The JSON formatted file is read, the events do get into ELK
2. When I turn on debugging in filebeat I get the following log entries prior to each record push:

2019-09-18T18:58:50.181-0700    DEBUG   [processors]    processing/processors.go:108    Fail to apply processor client{drop_fields=json.actions, rename=[{From:json To:zeek.notice} {From:zeek.notice.src To:source.address} {From:zeek.notice.dst To:destination.address} {From:zeek.notice.uid To:zeek.session_id} {From:zeek.notice.p To:destination.port} {From:zeek.notice.conn To:zeek.notice.connnection_id} {From:zeek.notice.iconn To:zeek.notice.icmp_id} {From:zeek.notice.id.orig_h To:source.address} {From:zeek.notice.id.orig_p To:source.port} {From:zeek.notice.id.resp_h To:destination.address} {From:zeek.notice.id.resp_p To:destination.port} {From:zeek.notice.proto To:network.transport} {From:zeek.notice.id.orig_p To:source.port} {From:zeek.notice.f.id To:zeek.notice.file.id} {From:zeek.notice.f.parent_id To:dzeek.notice.file.parent_id} {From:zeek.notice.f.source To:zeek.notice.file.source} {From:zeek.notice.f.is_orig To:zeek.notice.file.is_orig} {From:zeek.notice.f.seen_bytes To:zeek.notice.file.seen_bytes} {From:zeek.notice.f.total_bytes To:zeek.notice.file.total_bytes} {From:zzeek.notice.file_mime_type To:zeek.notice.file.mime_type}], drop_fields=zeek.notice.remote_location, zeek.notice.f, community_id=[target=network.community_id, fields=[source_ip=source.address, source_port=source.port, destination_ip=destination.address, destination_port=destination.port, transport_protocol=network.transport, icmp_type=icmp.type, icmp_code=icmp.code], seed=0]}: key not found
2019-09-18T18:58:50.181-0700    DEBUG   [rename]        actions/rename.go:81    Failed to rename fields in processor: target field source.port already exists, drop or rename this field first
2019-09-18T18:58:50.181-0700    DEBUG   [processors]    processing/processors.go:108    Fail to apply processor client{drop_fields=json.actions, rename=[{From:json To:zeek.notice} {From:zeek.notice.src To:source.address} {From:zeek.notice.dst To:destination.address} {From:zeek.notice.uid To:zeek.session_id} {From:zeek.notice.p To:destination.port} {From:zeek.notice.conn To:zeek.notice.connnection_id} {From:zeek.notice.iconn To:zeek.notice.icmp_id} {From:zeek.notice.id.orig_h To:source.address} {From:zeek.notice.id.orig_p To:source.port} {From:zeek.notice.id.resp_h To:destination.address} {From:zeek.notice.id.resp_p To:destination.port} {From:zeek.notice.proto To:network.transport} {From:zeek.notice.id.orig_p To:source.port} {From:zeek.notice.f.id To:zeek.notice.file.id} {From:zeek.notice.f.parent_id To:dzeek.notice.file.parent_id} {From:zeek.notice.f.source To:zeek.notice.file.source} {From:zeek.notice.f.is_orig To:zeek.notice.file.is_orig} {From:zeek.notice.f.seen_bytes To:zeek.notice.file.seen_bytes} {From:zeek.notice.f.total_bytes To:zeek.notice.file.total_bytes} {From:zzeek.notice.file_mime_type To:zeek.notice.file.mime_type}], drop_fields=zeek.notice.remote_location, zeek.notice.f, community_id=[target=network.community_id, fields=[source_ip=source.address, source_port=source.port, destination_ip=destination.address, destination_port=destination.port, transport_protocol=network.transport, icmp_type=icmp.type, icmp_code=icmp.code], seed=0]}: key not found, key not found

3. Again I'm definitely new to this, but it looks like it might be having a problem with renaming fields? I see entries for geoip in the pipeline.json file for the zeek module in filebeat.

      "set": {
        "field": "source.ip",
        "value": "{{source.address}}"
      }
    },
    {
      "set": {
        "field": "destination.ip",
        "value": "{{destination.address}}"
      }
    },
...
    {
      "geoip": {
        "field": "destination.ip",
        "target_field": "destination.geo"
      }
    },
    {
      "geoip": {
        "field": "source.ip",
        "target_field": "source.geo"
      }
    }

Am I correct that this is preventing geo from functioning? Can anyone help point me in the right direction as to why it isn't finding whatever key it is referring to?

Thanks!

[0x00]

I also just found this in the connection.yml file which appears to correspond with some of the error message entries in the prior message, perhaps this will help...

processors:
  - drop_fields:
      fields: ["json.orig_bytes","json.resp_bytes","json.tunnel_parents"]
  - rename:
      fields:
        - from: "json"
          to: "zeek.connection"

        - from: "zeek.connection.duration"
          to: "temp.duration"

        - from: "zeek.connection.id.orig_h"
          to: "source.address"

        - from: "zeek.connection.id.orig_p"
          to: "source.port"

        - from: "zeek.connection.id.resp_h"
          to: "destination.address"

        - from: "zeek.connection.id.resp_p"
          to: "destination.port"

        - from: "zeek.connection.proto"
          to: "network.transport"

        - from: "zeek.connection.service"
          to: "network.application"

        - from: "zeek.connection.uid"
          to: "zeek.session_id"

        - from: "zeek.connection.orig_ip_bytes"
          to: "source.bytes"

        - from: "zeek.connection.resp_ip_bytes"
          to: "destination.bytes"

        - from: "zeek.connection.orig_pkts"
          to: "source.packets"

        - from: "zeek.connection.resp_pkts"
          to: "destination.packets"

        - from: "zeek.connection.conn_state"
          to: "zeek.connection.state"

      ignore_missing: true
      fail_on_error: false