# Zeek Import - Timestamp & Geo Failing **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 8, 2019, 5:49am UTC](https://discuss.elastic.co/t/zeek-import-timestamp-geo-failing/198536 "2019-09-08T05:49:13Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![0x00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/0x00/32/52748_2.png) [@0x00](https://discuss.elastic.co/u/0x00)\ **Post date:** [September 8, 2019, 5:49am UTC](https://discuss.elastic.co/t/zeek-import-timestamp-geo-failing/198536/1 "2019-09-08T05:49:13Z")
I am attempting to use Filebeat on Ubuntu Linux to import a JSON formatted conn.log as the obfuscated sample is shown below: `{"_path":"conn","_system_name":"sensor","_write_ts":"2019-07-02T15:53:03.511364Z","ts":"2019-07-02T15:52:46.377889Z","uid":"CByC0qkmPzrL8w4Akj","id.orig_h":"12.34.56.78","id.orig_p":64069,"id.resp_h":"98.76.54.32","id.resp_p":135,"proto":"tcp","service":"dce_rpc","duration":12.133461,"orig_bytes":2355,"resp_bytes":395,"conn_state":"SF","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ShADadFf","orig_pkts":9,"orig_ip_bytes":2727,"resp_pkts":7,"resp_ip_bytes":687,"tunnel_parents":[],"orig_l2_addr":"00:11:22:33:44:55","resp_l2_addr":"aa:bb:cc:dd:ee:ff"}` I have the zeek module enabled: ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/6/268acf6c5a6dfefbb34de1a08f7a8f7e68b5cff5.png) And it successfully imports the log. ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/a/ea94299a0a0bcbb593470f253c7c1ed4e82a8279.png) However it does "NOT" allow me to use the proper, original, timestamp and instead uses the time listed when the log was processed. As an additional challenge, it doesn't seem to be adding geo location. How can I go about getting it to use the right (original conn log time) timestamp and get it to process geo location data against the conn log? Can I update `fields.yml` or `connection.yml` or something of this sort? ---
**Author:** ![0x00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/0x00/32/52748_2.png) [@0x00](https://discuss.elastic.co/u/0x00)\ **Post date:** [September 8, 2019, 6:21am UTC](https://discuss.elastic.co/t/zeek-import-timestamp-geo-failing/198536/2 "2019-09-08T06:21:29Z")
I tried adding this to `/var/usr/filebeat/module/zeek/connection/ingest/pipeline.json` which did not help. ``` { "rename": { "field": "@timestamp", "target_field": "event.created" } }, { "date": { "field": "zeek.notice.ts", "target_field": "@timestamp", "formats": ["yyyy-MM-dd HH:mm:ss"] } }, ``` I tried to add this to `/etc/filebeat/fields.yml` which also did not help. ``` - name: connection.ts type: date description: > my timestamp field ``` And I tried adding this to `/var/usr/filebeat/module/zeek/connection/config/connection.yml` which clearly also failed. ``` - from: "zeek.connection.ts" to: "event.created" ``` Please forgive my naivety around how this is all set up. I am still in the process of learning all of this and trying to figure out how things still go together. ---
**Author:** ![0x00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/0x00/32/52748_2.png) [@0x00](https://discuss.elastic.co/u/0x00)\ **Post date:** [September 19, 2019, 2:33am UTC](https://discuss.elastic.co/t/zeek-import-timestamp-geo-failing/198536/3 "2019-09-19T02:33:11Z")
I am still fighting with this but have an update. **Timestamp Resolution (_sort of_):** I switched from having filebeat send the data straight to elasticsearch and instead started sending it to logstash. **This action helped resolve the timestamp problem and I can now select it** though I still don't understand why I couldn't do that when sending the data `JSON -> Filebeat -> Elasticsearch`. However, _geo / geo point / geoip_ is still not working and I'm fighting with how to get it functioning properly. 1. The JSON formatted file is read, the events do get into ELK 2. When I turn on debugging in filebeat I get the following log entries prior to each record push: ```auto 2019-09-18T18:58:50.181-0700 DEBUG [processors] processing/processors.go:108 Fail to apply processor client{drop_fields=json.actions, rename=[{From:json To:zeek.notice} {From:zeek.notice.src To:source.address} {From:zeek.notice.dst To:destination.address} {From:zeek.notice.uid To:zeek.session_id} {From:zeek.notice.p To:destination.port} {From:zeek.notice.conn To:zeek.notice.connnection_id} {From:zeek.notice.iconn To:zeek.notice.icmp_id} {From:zeek.notice.id.orig_h To:source.address} {From:zeek.notice.id.orig_p To:source.port} {From:zeek.notice.id.resp_h To:destination.address} {From:zeek.notice.id.resp_p To:destination.port} {From:zeek.notice.proto To:network.transport} {From:zeek.notice.id.orig_p To:source.port} {From:zeek.notice.f.id To:zeek.notice.file.id} {From:zeek.notice.f.parent_id To:dzeek.notice.file.parent_id} {From:zeek.notice.f.source To:zeek.notice.file.source} {From:zeek.notice.f.is_orig To:zeek.notice.file.is_orig} {From:zeek.notice.f.seen_bytes To:zeek.notice.file.seen_bytes} {From:zeek.notice.f.total_bytes To:zeek.notice.file.total_bytes} {From:zzeek.notice.file_mime_type To:zeek.notice.file.mime_type}], drop_fields=zeek.notice.remote_location, zeek.notice.f, community_id=[target=network.community_id, fields=[source_ip=source.address, source_port=source.port, destination_ip=destination.address, destination_port=destination.port, transport_protocol=network.transport, icmp_type=icmp.type, icmp_code=icmp.code], seed=0]}: key not found 2019-09-18T18:58:50.181-0700 DEBUG [rename] actions/rename.go:81 Failed to rename fields in processor: target field source.port already exists, drop or rename this field first 2019-09-18T18:58:50.181-0700 DEBUG [processors] processing/processors.go:108 Fail to apply processor client{drop_fields=json.actions, rename=[{From:json To:zeek.notice} {From:zeek.notice.src To:source.address} {From:zeek.notice.dst To:destination.address} {From:zeek.notice.uid To:zeek.session_id} {From:zeek.notice.p To:destination.port} {From:zeek.notice.conn To:zeek.notice.connnection_id} {From:zeek.notice.iconn To:zeek.notice.icmp_id} {From:zeek.notice.id.orig_h To:source.address} {From:zeek.notice.id.orig_p To:source.port} {From:zeek.notice.id.resp_h To:destination.address} {From:zeek.notice.id.resp_p To:destination.port} {From:zeek.notice.proto To:network.transport} {From:zeek.notice.id.orig_p To:source.port} {From:zeek.notice.f.id To:zeek.notice.file.id} {From:zeek.notice.f.parent_id To:dzeek.notice.file.parent_id} {From:zeek.notice.f.source To:zeek.notice.file.source} {From:zeek.notice.f.is_orig To:zeek.notice.file.is_orig} {From:zeek.notice.f.seen_bytes To:zeek.notice.file.seen_bytes} {From:zeek.notice.f.total_bytes To:zeek.notice.file.total_bytes} {From:zzeek.notice.file_mime_type To:zeek.notice.file.mime_type}], drop_fields=zeek.notice.remote_location, zeek.notice.f, community_id=[target=network.community_id, fields=[source_ip=source.address, source_port=source.port, destination_ip=destination.address, destination_port=destination.port, transport_protocol=network.transport, icmp_type=icmp.type, icmp_code=icmp.code], seed=0]}: key not found, key not found ``` 1. Again I'm definitely new to this, but it looks like it might be having a problem with renaming fields? I see entries for geoip in the `pipeline.json` file for the zeek module in filebeat. ```auto "set": { "field": "source.ip", "value": "{{source.address}}" } }, { "set": { "field": "destination.ip", "value": "{{destination.address}}" } }, ... { "geoip": { "field": "destination.ip", "target_field": "destination.geo" } }, { "geoip": { "field": "source.ip", "target_field": "source.geo" } } ``` Am I correct that this is preventing geo from functioning? Can anyone help point me in the right direction as to why it isn't finding whatever key it is referring to? Thanks! ---
**Author:** ![0x00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/0x00/32/52748_2.png) [@0x00](https://discuss.elastic.co/u/0x00)\ **Post date:** [September 19, 2019, 2:37am UTC](https://discuss.elastic.co/t/zeek-import-timestamp-geo-failing/198536/4 "2019-09-19T02:37:52Z")
I also just found this in the `connection.yml` file which appears to correspond with some of the error message entries in the prior message, perhaps this will help... ```auto processors: - drop_fields: fields: ["json.orig_bytes","json.resp_bytes","json.tunnel_parents"] - rename: fields: - from: "json" to: "zeek.connection" - from: "zeek.connection.duration" to: "temp.duration" - from: "zeek.connection.id.orig_h" to: "source.address" - from: "zeek.connection.id.orig_p" to: "source.port" - from: "zeek.connection.id.resp_h" to: "destination.address" - from: "zeek.connection.id.resp_p" to: "destination.port" - from: "zeek.connection.proto" to: "network.transport" - from: "zeek.connection.service" to: "network.application" - from: "zeek.connection.uid" to: "zeek.session_id" - from: "zeek.connection.orig_ip_bytes" to: "source.bytes" - from: "zeek.connection.resp_ip_bytes" to: "destination.bytes" - from: "zeek.connection.orig_pkts" to: "source.packets" - from: "zeek.connection.resp_pkts" to: "destination.packets" - from: "zeek.connection.conn_state" to: "zeek.connection.state" ignore_missing: true fail_on_error: false ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 17, 2019, 2:38am UTC](https://discuss.elastic.co/t/zeek-import-timestamp-geo-failing/198536/5 "2019-10-17T02:38:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.