Filebeat 6.0.0 prospectors and fields

[panic]

Hi,
I'm currently using filebeat 5.6.5 to extract data from specific log types with using the following configuration:

filebeat.prospectors:

- type: log
  paths:
    - /logfiles/x.log
  fields:
     document_type: x
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

- type: log
  paths:
    - /logfiles/y.log
  fields:
     document_type: y

- type: log
  paths:
    - /logfiles/z.log
  fields:
     document_type: z

The output in 5.6.5 looks something like this:

{"@timestamp":"2017-12-07T03:37:45.936Z","beat":{"hostname":"aaa","name":"aaa","version":"5.6.5"},"fields":{"document_type":"x"},"source":"/logfiles/x.log"}
{"@timestamp":"2017-12-07T03:37:45.936Z","beat":{"hostname":"aaa","name":"aaa","version":"5.6.5"},"fields":{"document_type":"y"},"source":"/logfiles/y.log"}
{"@timestamp":"2017-12-07T03:37:45.936Z","beat":{"hostname":"aaa","name":"aaa","version":"5.6.5"},"fields":{"document_type":"z"},"source":"/logfiles/z.log"}

Using the same configuration in 6.0.0 I end up with all of the document_type fields being set as "z" for all unique sources. (also the output is a lot messier... the fields aren't necessarily in the same order)

{"@timestamp":"2017-12-07T01:19:48.018Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"source":"/logfiles/x.log","prospector":{"type":"log"},"fields":{"document_type":"z"},"beat":{"name":"aaa","hostname":"aaa","version":"6.0.0"}}
{"@timestamp":"2017-12-07T01:19:48.018Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"source":"/logfiles/y.log,"prospector":{"type":"log"},"fields":{"document_type":"z"},"beat":{"name":"aaa","hostname":"aaa","version":"6.0.0"}}
{"@timestamp":"2017-12-07T01:19:48.018Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"prospector":{"type":"log"},"fields":{"document_type":"z"},"beat":{"version":"6.0.0","name":"aaa","hostname":"aaa"},"source":"/logfiles/z.log}

It just seems to set the last value... I believe this is a bug?
Has something changed in the way fields are handled?
I looked up the reference, but from what I can tell, it should work.

[ruflin]

You are hitting this issue here: https://github.com/elastic/beats/pull/5808

[panic]

Ah, thanks ruflin!