6.0.0-rc1: json.overwrite_keys not working with @timestamp

Elastic Stack Beats
1

Hi,

we are trying to forward all json messages in a log file to logstash using Filebeat, but the timestamps are already off on the Filebeat side. With 5.0.1 it is working, with 6.0.0-rc1 we get duplicate @timestamp fields.

Sample JSON:
{ "@timestamp": "2017-10-17T10:03:14.301Z", "request": "/" }

Sample Filebeat Config 5.0.1 (working):

filebeat.prospectors:
- input_type: log
  paths:
    - serverlogs/apache.json
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true
  fields_under_root: true

output.console:
  pretty: true

Sample Filebeat Config 6.0.0-rc1 (not working):

filebeat.prospectors:
- prospector_type: log
  paths:
    - serverlogs/apache.json
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true
  fields_under_root: true

output.console:
  pretty: true

Running with a 5.0.1 installation (download, untar, add serverlogs/apache.json file with sample provided above, add filebeat.json.yml with sample provided above, run):

[friedreb@pc64901 filebeat-5.0.1-linux-x86_64]$ rm -rf data/ && ./filebeat -c filebeat.json.yml
{
  "@timestamp": "2017-10-17T10:03:14.301Z",
  "beat": {
    "hostname": "pc64901",
    "name": "pc64901",
    "version": "5.0.1"
  },
  "input_type": "log",
  "offset": 61,
  "request": "/",
  "source": "serverlogs/apache.json",
  "type": "log"
}

Running with a 6.0.0-rc1 installation (download, untar, add serverlogs/apache.json file with sample provided above, add filebeat.json.yml with sample provided above, run):

[friedreb@pc64901 filebeat-6.0.0-rc1-linux-x86_64]$ ./filebeat -c filebeat.json.yml
{
  "@timestamp": "2017-10-25T07:54:57.673Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.0.0-rc1"
  },
  "@timestamp": "2017-10-17T10:03:14.301Z",
  "beat": {
    "name": "pc64901",
    "hostname": "pc64901",
    "version": "6.0.0-rc1"
  },
  "source": "/products/filebeat-6.0.0-rc1-linux-x86_64/serverlogs/apache.json",
  "offset": 61,
  "request": "/"
}

You can clearly see that the 6.0.0-rc1 installation contains duplicate @timestamp fields. This is resulting in wrong timestamps.

I guess this is a bug in 6.0.0?

Another interesting behavior is that on 5.0.1 the json line is imported only once and with every other run of filebeat it is doing nothing (no new data). With 6.0.0-rc1 the json line is parsed every time I start filebeat. I guess this is another bug?

Thanks & Best regards,
Bernhard Friedreich

2

Version 5.6.3 (latest GA) doesn't have the timestamp problem. The json.overwrite_keys is working and I get the correct timestamp. So the problem seems to have been introduced with the 6.0 branch.

BUT: it also has the same behavior (as 6.0.0-rc1) in that it re-reads (and parses) the logfile every time.

3

I was able to reproduce your issue and it does seem like a bug. Could you please open an issue on Github?

4

Also, please add "Pioneer Program" label to your issue, so you could be recognized for your help.

5

My bad. You might not have the rights to add this label. But we will add it, after you opened the issue.

6

Thank you for the fast reply! :slight_smile:
I've created the issue for the json.overwrite_keys not working: https://github.com/elastic/beats/issues/5440
Please add the appropriate labels :slight_smile: and thanks for the "Pioneer Program" :slight_smile:

Shall I create a separate issue for the json being parsed every time I restart filebeat?

7

Yes, please. :slight_smile:

8

Done :slight_smile:

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.