# FileBeat Processing @timestamp instead of log @timestamp

**URL:** <https://discuss.elastic.co/t/filebeat-processing-timestamp-instead-of-log-timestamp/216979>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 29, 2020, 10:21am UTC](https://discuss.elastic.co/t/filebeat-processing-timestamp-instead-of-log-timestamp/216979 "2020-01-29T10:21:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![salunkerahul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/salunkerahul/32/47522_2.png) [@salunkerahul](https://discuss.elastic.co/u/salunkerahul)\
**Post date:** [January 29, 2020, 10:21am UTC](https://discuss.elastic.co/t/filebeat-processing-timestamp-instead-of-log-timestamp/216979/1 "2020-01-29T10:21:54Z")

</div>

I'm using the following flow _ **FileBeat-\>Elastic-\>Kibana** _ on Windows-7 using v7.5.2 and the timestamp as seen in the Kibana _Discover_ window always corresponds to the time the log was processed rather than represent the log's timestamp. The log looks as below:

```
{"stored_proc_name":"sp_prepexecrpc","@timestamp":"2020-01-29T09:51:30.2298715","duration":216,"client_app_name":"TEST","sql_server":"DB01","database":"Cust_R"}
{"stored_proc_name":"sp_prepexecrpc","@timestamp":"2020-01-29T09:51:30.2306409","duration":221,"client_app_name":"TEST","sql_server":"DB01","database":"Cust_R"}

```

**filebeat.yml**

```
filebeat.inputs:
- type: log

  enabled: true

  paths:
- 'D:\Scratch\C#\T3\Monitor.SP_Executions\bin\Debug\*.log'
# - 'D:\Scratch\C#\T2\LiveStreaming\bin\Debug\*.log'

json.keys_under_root: true
json.overwrite_keys: true

processors:
- decode_json_fields:
fields: ["message"]
max_depth: 1
process_array: true
add_error_key: true
overwrite_keys: true

```

From FileBeat.log file I see that the timestamp is not being updated to use the time stamp from my log

```
2020-01-29T10:10:44.758Z	DEBUG	[processors]	processing/processors.go:186	Publish event: {
  "@timestamp": "2020-01-29T10:10:44.758Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.2"
  },
  "host": {
    "id": "75bb063d-5ad1-4c5a-9fb0-db3b045fcfc5",
    "hostname": "EUD2362",
    "architecture": "x86_64",
    "os": {
      "name": "Windows 7 Enterprise",
      "kernel": "6.1.7601.24540 (win7sp1_ldr_escrow.191127-1706)",
      "build": "7601.24533",
      "platform": "windows",
      "version": "6.1",
      "family": "windows"
    },
    "name": "EUD2362"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "26200367-10ad-4566-8f12-c38dd987f4b2",
    "hostname": "D2362",
    "id": "7cf1c15a-0903-4551-8dd7-4f21947ed61f",
    "version": "7.5.2"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "log": {
    "offset": 15906770,
    "file": {
      "path": "D:\\Scratch\\C#\\T3\\Monitor.SP_Executions\\bin\\Debug\\STGSQL05_SP_EXECUTIONS.log"
    }
  },
  "message": "{\"stored_proc_name\":\"sp_prepexecrpc\",\"@timestamp\":\"2020-01-29T09:45:31.9330272\",\"duration\":292,\"client_app_name\":\"TEST\",\"sql_server\":\"DB01\",\"database\":\"Cust_R\"}",
  "input": {
    "type": "log"
  }
}

```

I toggled **json.overwrite\_keys: true** this did not help. i.e., after toggling I made sure I **stopped** and **started** filebeat

I'm a newbie to ELK and I believe I'm missing something obvious. Any help in the right direction is much appreciated.  
Also, what is the recommended practice for **filebeat.yml** changes to be visible?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [January 29, 2020, 1:25pm UTC](https://discuss.elastic.co/t/filebeat-processing-timestamp-instead-of-log-timestamp/216979/2 "2020-01-29T13:25:39Z")

</div>

Hi!

Two problems here:

1. First of all your configuration seems to not be working since the json messages are not parsed at all. Here is what I use:

```auto
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /tmp/filebeattest
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true

```

1. The timestamp format looks problematic (not supported) I changed that to:

```auto
{"stored_proc_name":"sp_prepexecrpc","@timestamp":"2020-01-28T13:05:13.672Z","duration":216,"client_app_name":"TEST","sql_server":"DB01","database":"Cust_R"}
{"stored_proc_name":"sp_prepexecrpc","@timestamp":"2020-01-29T09:51:30.230Z","duration":221,"client_app_name":"TEST","sql_server":"DB01","database":"Cust_R"}

```

Then the output I see looks like:

```auto
{
  "@timestamp": "2020-01-28T13:05:13.672Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.0.0"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/tmp/filebeattest"
    }
  },
  "input": {
    "type": "log"
  },
  "host": {
    "name": "local"
  },
  "agent": {
    "version": "8.0.0",
    "type": "filebeat",
    "ephemeral_id": "f3f9fab7-3a44-4182-b43d-5f07fba147fa",
    "hostname": "local",
    "id": "368f6f6b-a6e6-4af2-94c8-72efe7e8ef55"
  },
  "stored_proc_name": "sp_prepexecrpc",
  "client_app_name": "TEST",
  "database": "Cust_R",
  "duration": 216,
  "sql_server": "DB01",
  "ecs": {
    "version": "1.4.0"
  }
}

```

Regarding the timestamp format issue, if you want to keep your current one I guess you should do go with something like what is proposed to `https://discuss.elastic.co/t/get-timestamp-from-log-lines/117816`.

---

<div class="post-metadata">

**Author:** ![salunkerahul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/salunkerahul/32/47522_2.png) [@salunkerahul](https://discuss.elastic.co/u/salunkerahul)\
**Post date:** [January 29, 2020, 2:20pm UTC](https://discuss.elastic.co/t/filebeat-processing-timestamp-instead-of-log-timestamp/216979/3 "2020-01-29T14:20:02Z")

</div>

Hi Chris,  
Thanks for the tip.  
It was indeed the wrong timestamp I had to specify explicitly to dump DateTime as UTC.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 26, 2020, 2:20pm UTC](https://discuss.elastic.co/t/filebeat-processing-timestamp-instead-of-log-timestamp/216979/4 "2020-02-26T14:20:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.