# Filebeat and logstash, merging logs and separating them

**URL:** <https://discuss.elastic.co/t/filebeat-and-logstash-merging-logs-and-separating-them/42582>\
**Category:** Beats\
**Created:** [February 24, 2016, 11:56am UTC](https://discuss.elastic.co/t/filebeat-and-logstash-merging-logs-and-separating-them/42582 "2016-02-24T11:56:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dan.henning](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@dan.henning](https://discuss.elastic.co/u/dan.henning)\
**Post date:** [February 24, 2016, 11:56am UTC](https://discuss.elastic.co/t/filebeat-and-logstash-merging-logs-and-separating-them/42582/1 "2016-02-24T11:56:29Z")

</div>

Hello, I have multiple log txt files that have two separate logs inside that span multiple lines each.  
Here is an example:

13:43:58 Message loaded: D:\mail\heMail\Drop\ \003.eml  
13:43:58 Attempting to process message as dropbox email.  
13:43:59 Message is not a dropbox email.  
13:43:59 Attempting to process message as dropbox context container email.  
13:43:59 Attempting to process message as dropbox document folder email.  
13:44:00 Failed to load user by email address: Object reference not set to an instance of an object.. Item will be processed as SysAdmin. Sender: [droptest@email.com](mailto:droptest@email.com)  
13:44:00 Message is not a dropbox document container email.  
13:44:00 Attempting to process message as dropbox task or discussion container email.  
13:44:00 Message is not a dropbox context container email.  
13:44:00 Attempting to process message as task or discussion response  
13:44:00 Failed to process message as task or discussion item. Must specify valid information for parsing in the string.  
14:31:32 Message loaded: D:\Mail\HeMail\Drop\ \000004.eml  
14:31:32 Attempting to process message as dropbox email.  
14:31:32 Message is not a dropbox email.  
14:31:32 Attempting to process message as dropbox context container email.  
14:31:32 Attempting to process message as dropbox document folder email.  
14:31:38 Can't create documents in this folder. It is not valid. ContainerID: bb5563. Email address is not valid  
14:31:39 Failed to load user by email address. Item will be processed as SysAdmin. Sender: [drop@email.com](mailto:drop@email.com)  
14:31:39 Failed to process recipients from the email message.  
14:31:39 Message is not a dropbox context container email.  
14:31:39 Attempting to process message as task or discussion response  
14:31:39 Failed to process message as task or discussion item. Must specify valid information for parsing in the string.

This is one txt file, I would like to have each log which starts with "Message loaded" list each line as one entry in Kibana instead of all the lines being separated out.

So in conclusion i need each log which starts with "Message loaded" to list all the lines of that log as one entry in kibana until it reaches the next log in the file which starts with the date and "Message loaded" and then this log as one entry also and so on.

Thanks hope you can help

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 24, 2016, 12:08pm UTC](https://discuss.elastic.co/t/filebeat-and-logstash-merging-logs-and-separating-them/42582/2 "2016-02-24T12:08:28Z")

</div>

That's not how one usually does things. Why would you want to merge log entries like that? And are you sure that the log messages from two messages can't be interleaved in the output?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [February 24, 2016, 7:10pm UTC](https://discuss.elastic.co/t/filebeat-and-logstash-merging-logs-and-separating-them/42582/3 "2016-02-24T19:10:02Z")

</div>

If I understand you correctly, you basically want to send all content (emails) of `*.eml` files to be send as a single event?

If all `eml`-files are put into the same directory, why not define a prospector on `D:\Mail\HeMail\Drop\*.eml` with multiline pattern basically merging all file content?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:55pm UTC](https://discuss.elastic.co/t/filebeat-and-logstash-merging-logs-and-separating-them/42582/4 "2017-07-05T21:55:25Z")

</div>


