# 6.0.0-rc1: json.overwrite\_keys not working with @timestamp

**URL:** <https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 25, 2017, 8:03am UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189 "2017-10-25T08:03:48Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![friesoft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/friesoft/32/23388_2.png) [@friesoft](https://discuss.elastic.co/u/friesoft)\
**Post date:** [October 25, 2017, 8:03am UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189/1 "2017-10-25T08:03:48Z")

</div>

Hi,

we are trying to forward all json messages in a log file to logstash using Filebeat, but the timestamps are already off on the Filebeat side. With 5.0.1 it is working, with 6.0.0-rc1 we get duplicate @timestamp fields.

Sample JSON:  
`{ "@timestamp": "2017-10-17T10:03:14.301Z", "request": "/" }`

Sample Filebeat Config 5.0.1 (working):

> ```
> filebeat.prospectors:
> - input_type: log
> paths:
> - serverlogs/apache.json
> json.keys_under_root: true
> json.add_error_key: true
> json.overwrite_keys: true
> fields_under_root: true
> 
> output.console:
> pretty: true
> 
> ```

Sample Filebeat Config 6.0.0-rc1 (not working):

> ```
> filebeat.prospectors:
> - prospector_type: log
> paths:
> - serverlogs/apache.json
> json.keys_under_root: true
> json.add_error_key: true
> json.overwrite_keys: true
> fields_under_root: true
> 
> output.console:
> pretty: true
> 
> ```

Running with a 5.0.1 installation (download, untar, add serverlogs/apache.json file with sample provided above, add filebeat.json.yml with sample provided above, run):

> ```
> [friedreb@pc64901 filebeat-5.0.1-linux-x86_64]$ rm -rf data/ && ./filebeat -c filebeat.json.yml
> {
> "@timestamp": "2017-10-17T10:03:14.301Z",
> "beat": {
> "hostname": "pc64901",
> "name": "pc64901",
> "version": "5.0.1"
> },
> "input_type": "log",
> "offset": 61,
> "request": "/",
> "source": "serverlogs/apache.json",
> "type": "log"
> }
> 
> ```

Running with a 6.0.0-rc1 installation (download, untar, add serverlogs/apache.json file with sample provided above, add filebeat.json.yml with sample provided above, run):

> ```
> [friedreb@pc64901 filebeat-6.0.0-rc1-linux-x86_64]$ ./filebeat -c filebeat.json.yml
> {
> "@timestamp": "2017-10-25T07:54:57.673Z",
> "@metadata": {
> "beat": "filebeat",
> "type": "doc",
> "version": "6.0.0-rc1"
> },
> "@timestamp": "2017-10-17T10:03:14.301Z",
> "beat": {
> "name": "pc64901",
> "hostname": "pc64901",
> "version": "6.0.0-rc1"
> },
> "source": "/products/filebeat-6.0.0-rc1-linux-x86_64/serverlogs/apache.json",
> "offset": 61,
> "request": "/"
> }
> 
> ```

You can clearly see that the 6.0.0-rc1 installation contains duplicate @timestamp fields. This is resulting in wrong timestamps.

I guess this is a bug in 6.0.0?

Another interesting behavior is that on 5.0.1 the json line is imported only once and with every other run of filebeat it is doing nothing (no new data). With 6.0.0-rc1 the json line is parsed every time I start filebeat. I guess this is another bug?

Thanks & Best regards,  
Bernhard Friedreich

---

<div class="post-metadata">

**Author:** ![friesoft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/friesoft/32/23388_2.png) [@friesoft](https://discuss.elastic.co/u/friesoft)\
**Post date:** [October 25, 2017, 8:39am UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189/2 "2017-10-25T08:39:21Z")

</div>

Version 5.6.3 (latest GA) doesn't have the timestamp problem. The json.overwrite\_keys is working and I get the correct timestamp. So the problem seems to have been introduced with the 6.0 branch.

BUT: it also has the same behavior (as 6.0.0-rc1) in that it re-reads (and parses) the logfile every time.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 25, 2017, 11:25am UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189/3 "2017-10-25T11:25:43Z")

</div>

I was able to reproduce your issue and it does seem like a bug. Could you please open an issue on Github?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 25, 2017, 12:46pm UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189/4 "2017-10-25T12:46:51Z")

</div>

Also, please add "Pioneer Program" label to your issue, so you could be recognized for your help.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 25, 2017, 12:50pm UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189/5 "2017-10-25T12:50:45Z")

</div>

My bad. You might not have the rights to add this label. But we will add it, after you opened the issue.

---

<div class="post-metadata">

**Author:** ![friesoft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/friesoft/32/23388_2.png) [@friesoft](https://discuss.elastic.co/u/friesoft)\
**Post date:** [October 25, 2017, 3:06pm UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189/6 "2017-10-25T15:06:42Z")

</div>

Thank you for the fast reply! 🙂  
I've created the issue for the json.overwrite\_keys not working: [https://github.com/elastic/beats/issues/5440](https://github.com/elastic/beats/issues/5440)  
Please add the appropriate labels 🙂 and thanks for the "Pioneer Program" 🙂

Shall I create a separate issue for the json being parsed every time I restart filebeat?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 25, 2017, 3:35pm UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189/7 "2017-10-25T15:35:51Z")

</div>

Yes, please. 🙂

---

<div class="post-metadata">

**Author:** ![friesoft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/friesoft/32/23388_2.png) [@friesoft](https://discuss.elastic.co/u/friesoft)\
**Post date:** [October 25, 2017, 8:01pm UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189/8 "2017-10-25T20:01:00Z")

</div>

Done 🙂

> <https://github.com/elastic/beats/issues/5442>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 22, 2017, 8:01pm UTC](https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189/9 "2017-11-22T20:01:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.