Filebeat field name collision with external prospectors?

[ceekay]

After working through issues with the new host object in Logstash errors after upgrading to filebeat-6.3.0 we've decided to go down the long road of namespacing our critical fields away from anything libbeat might put in the root in future.

However we've run in to an issue where nested fields with the same parent can't exist in both the main filebeat config and external prospector fragments.

E.g., using test_namespace as the parent object:

/etc/filebeat/filebeat.yml

filebeat:
  registry_file: /var/lib/filebeat/registry
  config.prospectors:
    enabled: true
    path: /etc/filebeat/conf.d/*.yml
fields:
  test_namespace.fqdn: civet.fqdn.xxxx
fields_under_root: true
output:
[...]

/etc/filebeat/conf.d/authlog.yml

- paths:
    - /var/log/auth.log
  type: log
  fields:
    test_namespace.container: syslog
  fields_under_root: true

Output

{
  "@timestamp": "2018-07-06T01:29:46.484Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.0"
  },
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "test_namespace": {
    "container": "syslog"
  },
  "beat": {
    "hostname": "civet",
    "version": "6.3.0",
    "name": "civet"
  },
  "host": {
    "name": "civet"
  },
  "source": "/var/log/auth.log",
  "offset": 67832,
  "message": "[redacted]"
}

Note that test_namespace.fqdn from filebeat.yml is missing from the output. If I rename the parent object in either of the configs, filebeat outputs both fields.

Is there any way to solve this, or is this a bug?

[adrisr (Adrian Serrano)]

It does look like a bug to me, I will raise the issue.

I've found an ugly workaround, but probably not scalable if you have a lot of external prospectors.

in your prospector (same as before):

fields:
  test_namespace.container: syslog
  fields_under_root: true

in the main fields filebeat.yml use a different name:

fields:
  test_namespace_fqdn: civet.fqdn.xxxx # note the _ instead of .
  fields_under_root: true

then use a rename processor to move it into the required namespace:

processors:
- rename:
    fields:
     - from: 'test_namespace_fqdn'
       to: 'test_namespace.fqdn'
    ignore_missing: false
    fail_on_error: true

[ceekay]

Thanks for the workaround, but if there's a fix coming, I'd rather wait. This is going to be packaged and automated so it all needs to be as clean as possible.

Do you have a link to the issue so I can subscribe to it?

[adrisr (Adrian Serrano)]

Here's the issue https://github.com/elastic/beats/issues/7528

We are working on a fix, thanks for your report