Filebeat field name collision with external prospectors?


After working through issues with the new host object in Logstash errors after upgrading to filebeat-6.3.0 we've decided to go down the long road of namespacing our critical fields away from anything libbeat might put in the root in future.

However we've run in to an issue where nested fields with the same parent can't exist in both the main filebeat config and external prospector fragments.

E.g., using test_namespace as the parent object:


  registry_file: /var/lib/filebeat/registry
    enabled: true
    path: /etc/filebeat/conf.d/*.yml
  test_namespace.fqdn: civet.fqdn.xxxx
fields_under_root: true


- paths:
    - /var/log/auth.log
  type: log
    test_namespace.container: syslog
  fields_under_root: true


  "@timestamp": "2018-07-06T01:29:46.484Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.0"
  "prospector": {
    "type": "log"
  "input": {
    "type": "log"
  "test_namespace": {
    "container": "syslog"
  "beat": {
    "hostname": "civet",
    "version": "6.3.0",
    "name": "civet"
  "host": {
    "name": "civet"
  "source": "/var/log/auth.log",
  "offset": 67832,
  "message": "[redacted]"

Note that test_namespace.fqdn from filebeat.yml is missing from the output. If I rename the parent object in either of the configs, filebeat outputs both fields.

Is there any way to solve this, or is this a bug?

(Adrian Serrano) #2

It does look like a bug to me, I will raise the issue.

I've found an ugly workaround, but probably not scalable if you have a lot of external prospectors.

in your prospector (same as before):

  test_namespace.container: syslog
  fields_under_root: true

in the main fields filebeat.yml use a different name:

  test_namespace_fqdn: civet.fqdn.xxxx # note the _ instead of .
  fields_under_root: true

then use a rename processor to move it into the required namespace:

- rename:
     - from: 'test_namespace_fqdn'
       to: 'test_namespace.fqdn'
    ignore_missing: false
    fail_on_error: true


Thanks for the workaround, but if there's a fix coming, I'd rather wait. This is going to be packaged and automated so it all needs to be as clean as possible.

Do you have a link to the issue so I can subscribe to it?

(Adrian Serrano) #4

Here's the issue

We are working on a fix, thanks for your report

(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.