Filebeat with multiline vs. multiline logstash codec


(Craig Foote) #1

I've seen some talk of a multiline option in filebeat but I'm in the process of setting up filebeat sending to logstash via beats input and need to parse multiline logs. Can I use the multiline codec in the input section for this?


(Magnus B├Ąck) #2

Yes, but only if Filebeat only monitors a single file. I strongly suggest that you use Filebeat's multiline feature (which will be available in Filebeat 1.1, slated for release next week but you can use a nightly build or build it yourself if you don't want to wait).


(Jerry Hoffmeister) #3

I stopped by to ask the same question. I have filebeat monitoring several files... is the multiline option a top level option or is / can it be per file?


(Jerry Hoffmeister) #4

nevermind - I've switched from the multiline filter in logstash to filebeat and it seems to be working just fine... kinda miss the grok support tho.


(Steffen Siering) #5

regarding grok the upcoming ingest-node feature might be of interest. E.g. see elasticsearch doc from master (development?) branch: https://www.elastic.co/guide/en/elasticsearch/reference/master/_grok_processor.html

filebeat's task is to collect and ship logs. Any kind of additional event processing/routing should be done by Logstash or upcoming Ingest Node in elasticsearch.


(Jerry Hoffmeister) #6

One new issue I see is when using the logstash multiline filter the log line got "multiline" added to the tags array. After switching to filebeat multiline, I no longer see the tag?


(ruflin) #7

See this issue here: https://github.com/elastic/beats/issues/957


(Vedran Maricevic) #8

It can be per file, in other words per prospector.


(system) #9