Filebeat and logstash, merging logs and separating them

Elastic Stack Beats
1

Hello, I have multiple log txt files that have two separate logs inside that span multiple lines each.
Here is an example:

13:43:58 Message loaded: D:\mail\heMail\Drop\ \003.eml
13:43:58 Attempting to process message as dropbox email.
13:43:59 Message is not a dropbox email.
13:43:59 Attempting to process message as dropbox context container email.
13:43:59 Attempting to process message as dropbox document folder email.
13:44:00 Failed to load user by email address: Object reference not set to an instance of an object.. Item will be processed as SysAdmin. Sender: droptest@email.com
13:44:00 Message is not a dropbox document container email.
13:44:00 Attempting to process message as dropbox task or discussion container email.
13:44:00 Message is not a dropbox context container email.
13:44:00 Attempting to process message as task or discussion response
13:44:00 Failed to process message as task or discussion item. Must specify valid information for parsing in the string.
14:31:32 Message loaded: D:\Mail\HeMail\Drop\ \000004.eml
14:31:32 Attempting to process message as dropbox email.
14:31:32 Message is not a dropbox email.
14:31:32 Attempting to process message as dropbox context container email.
14:31:32 Attempting to process message as dropbox document folder email.
14:31:38 Can't create documents in this folder. It is not valid. ContainerID: bb5563. Email address is not valid
14:31:39 Failed to load user by email address. Item will be processed as SysAdmin. Sender: drop@email.com
14:31:39 Failed to process recipients from the email message.
14:31:39 Message is not a dropbox context container email.
14:31:39 Attempting to process message as task or discussion response
14:31:39 Failed to process message as task or discussion item. Must specify valid information for parsing in the string.

This is one txt file, I would like to have each log which starts with "Message loaded" list each line as one entry in Kibana instead of all the lines being separated out.

So in conclusion i need each log which starts with "Message loaded" to list all the lines of that log as one entry in kibana until it reaches the next log in the file which starts with the date and "Message loaded" and then this log as one entry also and so on.

Thanks hope you can help

2

That's not how one usually does things. Why would you want to merge log entries like that? And are you sure that the log messages from two messages can't be interleaved in the output?

3

If I understand you correctly, you basically want to send all content (emails) of *.eml files to be send as a single event?

If all eml-files are put into the same directory, why not define a prospector on D:\Mail\HeMail\Drop\*.eml with multiline pattern basically merging all file content?