Filebeat multiline question

Elastic Stack Beats
1

My application maintains its state by persisting some info on a file within an interval of time. This file is not more than 10 lines at a time.

The format of the data that's inside the file would be something like this,

INFO=Hello World
DATE=Feb 10 2016 06:12:34
APPLICATION=TESTAPP
USER=BLAH

It seems like filebeat is only able to tail the file and read line by line which is not what I want. I want to parse these lines as a single JSON document and send to elastic.

Is there anyway to do so in filebeat ? Thank you for any input....

2

Filebeat has a multiline option for merging messages spread over multiple lines. Does a logical message always begin with an "INFO=" line, or how do you recognize the start of a message?

Filebeat won't able to parse the key/value pairs in the resulting message into JSON. You need Logstash for that.

3

Thank you for replying. Yes it always starts with INFO=

4

Okay, so the multiline configuration would be "unless the line starts with INFO=, join the current line with the previous line".

5

If I understand the logging strategy correctly, a new file is written for basically every message. That is @mregmi always wants to create one events per file. In this case an always match pattern should do the trick.

6

If I understand the logging strategy correctly, a new file is written for basically every message.

Yes, you're right.

7

Thank you for your inputs. I was able to create it as a single event and getting indexed into es.