# Multiple line and fully unstructured log file **URL:** \ **Category:** Logstash\ **Created:** [August 9, 2019, 12:59pm UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616 "2019-08-09T12:59:07Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Mohammad.ali](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammad.ali/32/45665_2.png) [@Mohammad.ali](https://discuss.elastic.co/u/Mohammad.ali)\ **Post date:** [August 9, 2019, 12:59pm UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616/1 "2019-08-09T12:59:07Z")
Hello my friends I got some logfiles that are fully unstructured what I am so far done is just for the first line of each log which are DATE HOUR TIMEZONE and some information %{DATE\_YMD:login\_date} %{TIME:login\_time} %{ISO8601\_TIMEZONE:login\_timezone} %{GREEDYDATA:unknow} (?\application-akka.\*) (?m)%{GREEDYDATA:unknown2} DATE\_YMD is pattern dir --→ DATE\_YMD %{YEAR}/%{MONTHNUM}/%{MONTHDAY} sometimes my log contains information about client login which is my problem because it goes to another line. The first question is how should I handle multiple line of a log and second one how to create a different pattern based on login(which can be success or error) 2016/08/05 15:03:40.445 +0430 - [INFO] - from application in application-akka.actor.default-dispatcher-1086 2013/08/05 19:03:40.445 +0430 - [INFO] - from application in application-akka.actor.default-dispatcher-1086 Login success - UserSession info : UserSession{userInfo=UserInfo{type='null', messageId='null', accountNumber='15391272736024 ', customerId='25814687', firstName='something', latinFirstName='null', lastNam='something', latinLastName='null', isCompany=false, bankAccounts=[BankAccount{id=-1, name='something', latinName='something'}], muserid='something', nationalCode='something', appuserId=null, email='something', onlineSessionTime=240, onlineSessionTimeMobile=5, allowSendSms=false, bourseAccountName='something'}, username='something', remoteAddress='something'} SupervisorActor , forwardToApi method with apiKey =2224 , Class = class something 2015/02/05 19:03:46.445 +0430 - [INFO] - from application in application-akka.actor.default-dispatcher-1086 Login error : {"title":"problem","description":"password is wrong","errorType":"BAD\_REQUEST","errorCode":4dddd0037,"UUID":"sdfsfsd66-jj-90"} ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 9, 2019, 1:18pm UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616/2 "2019-08-09T13:18:33Z")
You can use a multiline codec to combine all the lines for one event ``` codec => multiline { pattern => '^[\d/]{10} [\d:\.]{12} \+\d{4} - ' negate => true what => "previous" auto_flush_interval => 1 } ``` Then use grok to grab the login status ``` grok { match => { "message" => ["^Login %{WORD:loginStatus}"] } } ``` ---
**Author:** ![Mohammad.ali](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammad.ali/32/45665_2.png) [@Mohammad.ali](https://discuss.elastic.co/u/Mohammad.ali)\ **Post date:** [August 9, 2019, 1:22pm UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616/3 "2019-08-09T13:22:05Z")
> [@Badger](#): > > ```auto > > ``` Thanks alot..i will try this could you explain what codec =\> multiline { pattern =\> '\[1\]{10} [\d:.]{12} +\d{4} - ' negate =\> true what =\> "previous" auto\_flush\_interval =\> 1 } does? * * * 1. \d/ ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 9, 2019, 2:23pm UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616/4 "2019-08-09T14:23:20Z")
That codec checks each line to see if it matches that regexp. The regexp matches the date at the beginning of some of the lines. If a line does not match (negate =\> true) then it is appended to the previous line (what =\> previous). So when it sees a line that starts with a date it starts collecting lines up to the next line that starts with a date and creates a single event from that set of lines. An event is not flushed until the next line that starts with a date is seen, or the auto\_flush\_interval times out. If auto\_flush\_interval was not set then you would not get an event for the last message in the file, since it is not followed by anything that can flush the event. ---
**Author:** ![Mohammad.ali](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammad.ali/32/45665_2.png) [@Mohammad.ali](https://discuss.elastic.co/u/Mohammad.ali)\ **Post date:** [August 9, 2019, 4:11pm UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616/5 "2019-08-09T16:11:36Z")
Thanks alot.. ---
**Author:** ![Mohammad.ali](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammad.ali/32/45665_2.png) [@Mohammad.ali](https://discuss.elastic.co/u/Mohammad.ali)\ **Post date:** [August 10, 2019, 6:15am UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616/6 "2019-08-10T06:15:27Z")
Badger i forgot to tell you that iam using filebeat to send my log to logstash filebeat.inputs: - type: log paths: - /var/log/app.log output.logstash: hosts: ["localhost:5050"] -- so on other side logstash is using beats input plugin input { beats { port =\> "5050" } } filter { grok { match =\> { "message" =\> "%{DATE\_YMD:login\_date} %{TIME:login\_time} %{ISO8601\_TIMEZONE:login\_timezone} %{GREEDYDATA} (?\application-akka.\*)"} } } output { elasticsearch { stdout {} } } As far as i red on elastic site using multi line codec while using file beats input plugin can cause problem.am i right??? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 10, 2019, 12:40pm UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616/7 "2019-08-10T12:40:40Z")
> [@Mohammad.ali](#): > > using multi line codec while using file beats input plugin can cause problem If you have multiple beats sending data then yes, it would be a problem. You would have to adapt that codec configuration to the beat syntax for multiline. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 7, 2019, 12:40pm UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616/8 "2019-09-07T12:40:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.