I've seen some talk of a multiline option in filebeat but I'm in the process of setting up filebeat sending to logstash via beats input and need to parse multiline logs. Can I use the multiline codec in the input section for this?
Yes, but only if Filebeat only monitors a single file. I strongly suggest that you use Filebeat's multiline feature (which will be available in Filebeat 1.1, slated for release next week but you can use a nightly build or build it yourself if you don't want to wait).
I stopped by to ask the same question. I have filebeat monitoring several files... is the multiline option a top level option or is / can it be per file?
nevermind - I've switched from the multiline filter in logstash to filebeat and it seems to be working just fine... kinda miss the grok support tho.
1 Like
regarding grok the upcoming ingest-node feature might be of interest. E.g. see elasticsearch doc from master (development?) branch: https://www.elastic.co/guide/en/elasticsearch/reference/master/_grok_processor.html
filebeat's task is to collect and ship logs. Any kind of additional event processing/routing should be done by Logstash or upcoming Ingest Node in elasticsearch.
One new issue I see is when using the logstash multiline filter the log line got "multiline" added to the tags array. After switching to filebeat multiline, I no longer see the tag?
See this issue here: https://github.com/elastic/beats/issues/957
It can be per file, in other words per prospector.