Multiple files, multiple outputs

Elastic Stack Beats
1

I've an application that produces three files, each with very different multiline configs. Because of the multiline processing I created three different beats inputs in logstash, all on their own port, each handling the multilines. How do I configure filebeat to output a file (document_type) to a specific output? I really dont want to run three instances of filebeat and was disappointed to see that the config directory option does allow Individual prospector/output file configs run with one instance.

Will the 1.1 version address this issue or is the multiline option course grained and not per file?

Thanks
Richard

2

Hi,

in filebeat 1.1 we will have multiline support per prospector. You can already try our nightly builds.

You're using the multiline codec or the multiline filter? One options might be to set the document_type in filebeat per prospector and use conditionals + multiline filter in Logstash. Logstash multiline (in most recent version) creates an ID per connection, so lines from different connections are not mixed up. No need to use different ports per connection.

I think having mutliline close to source is the best solution. If you want to try 1.1 nightly build, you can find some documentation in the development branch.

3

Thanks, I've tried the latest nightly and it does indeed resolve my issue and works very well. Much better to have the multiline per prospector as its clear on its behaviour. Two of my file formats have been easy however the output from a Tanuki Wrapper service is a problem. Tanuki has a prefix which it applies to every line, including the wrapped multiline. An example being:

INFO   | jvm 1    | 2016/01/12 16:07:57 | Debug options:
INFO   | jvm 1    | 2016/01/12 16:07:57 |     file:/D:/SoftwareAG/profiles/IS/bin/.options loaded

It appears that wrapped lines are indented within the message section. Any suggestions as to how this is parsable with the new multiline? A pattern would detect the line continuation but how would the non-required prefix on subsequent lines be removed?

Thanks