Filebeat - Should metadata overwrite event data?

[jaquisar1 (Jaqui)]

Hi!

I'm having an issue with Filebeat 6.3.2, basically because of: https://github.com/elastic/beats/pull/7051 and I'm thinking weather this is a bug, a feature or just an inconvenience.

Basically, my input configuration is:

- type: log
  enabled: true
  paths:
    - /var/log/some-json.log
  fields_under_root: true
  json:
    keys_under_root: true
    overwrite_keys: true
    add_error_key: true

And each of the entries looks like this:

{ "fieldA": "A", "fieldB": "B", "host": "somehostname" }

Because of the change in pull #7051, the final event looks like this:

{ "fieldA": "A", "fieldB": "B", "host": { "name": "hostnameoffilebeat" } }

I understand using metadata inside "beat" field, but using a root "host" gets in the way of many types event which already contains the host (I'm thinking of nginx logs for example).

Looking at the code seems to be on purpose: https://github.com/elastic/beats/pull/5149

My opinion is that metadata fields should never overwrite event fields when they exists.

What do you guys think, does this deserve an issue in the repo or should I just use a workaround.

Edit: I've been searching and it looks like it's the issue described here: Logstash errors after upgrading to filebeat-6.3.0. Anyway, my question still remains, should metadata overwrite fields? My opinion is that it shouldn't...

Thanks.

[kvch (Noémi Ványi)]

Could you please open an issue on Github? https://github.com/elastic/beats/issues/new
This seems to be a bug as the documentation clearly states that if overwrite_keys is set, the JSON object overwrites the fields that Filebeat normally adds. (See in: https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-log.html#filebeat-input-log-config-json)