Hi,
I'm currently using filebeat 5.6.5 to extract data from specific log types with using the following configuration:
filebeat.prospectors:
- type: log
paths:
- /logfiles/x.log
fields:
document_type: x
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
- type: log
paths:
- /logfiles/y.log
fields:
document_type: y
- type: log
paths:
- /logfiles/z.log
fields:
document_type: z
The output in 5.6.5 looks something like this:
{"@timestamp":"2017-12-07T03:37:45.936Z","beat":{"hostname":"aaa","name":"aaa","version":"5.6.5"},"fields":{"document_type":"x"},"source":"/logfiles/x.log"}
{"@timestamp":"2017-12-07T03:37:45.936Z","beat":{"hostname":"aaa","name":"aaa","version":"5.6.5"},"fields":{"document_type":"y"},"source":"/logfiles/y.log"}
{"@timestamp":"2017-12-07T03:37:45.936Z","beat":{"hostname":"aaa","name":"aaa","version":"5.6.5"},"fields":{"document_type":"z"},"source":"/logfiles/z.log"}
Using the same configuration in 6.0.0 I end up with all of the document_type fields being set as "z" for all unique sources. (also the output is a lot messier... the fields aren't necessarily in the same order)
{"@timestamp":"2017-12-07T01:19:48.018Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"source":"/logfiles/x.log","prospector":{"type":"log"},"fields":{"document_type":"z"},"beat":{"name":"aaa","hostname":"aaa","version":"6.0.0"}}
{"@timestamp":"2017-12-07T01:19:48.018Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"source":"/logfiles/y.log,"prospector":{"type":"log"},"fields":{"document_type":"z"},"beat":{"name":"aaa","hostname":"aaa","version":"6.0.0"}}
{"@timestamp":"2017-12-07T01:19:48.018Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"prospector":{"type":"log"},"fields":{"document_type":"z"},"beat":{"version":"6.0.0","name":"aaa","hostname":"aaa"},"source":"/logfiles/z.log}
It just seems to set the last value... I believe this is a bug?
Has something changed in the way fields are handled?
I looked up the reference, but from what I can tell, it should work.