# Grok filter cannot parse log lines that are AFTER a multiline event **URL:** \ **Category:** Logstash\ **Created:** [April 4, 2016, 3:20pm UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255 "2016-04-04T15:20:26Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![pantheo](https://avatars.discourse-cdn.com/v4/letter/p/9de0a6/32.png) [@pantheo](https://discuss.elastic.co/u/pantheo)\ **Post date:** [April 4, 2016, 3:20pm UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255/1 "2016-04-04T15:20:26Z")
Hello everyone, I have a really annoying problem. As I mention in the title if I have an single-line event after a multiline event Logstash just freezes and cannot parse the rest of the file. However, If that same single line is before the multiline event then Logstash has no problem parsing it. My conf file is: ``` input { file { sincedb_path => "/dev/null/" # For debugging only! path => "/home/pantheo/Documents/ELK-playground/etenderingError-test.log" start_position => "beginning" codec => multiline { patterns_dir => ["./patterns"] pattern => "^%{LINE_BEGIN}" negate => true what => "previous" } } } filter { if "multiline" in [tags]{ grok { patterns_dir => ["./patterns"] match => {"message" => "%{LINE_BEGIN:start-of-line} %{CUSTOM_TIMESTAMP:the-time} %{THREAD_AGENT:agent} %{USER:user-ID} %{REQUEST_ID:request-ID} %{STATUS:status} %{DATA:message-body}\n((?m)%{GREEDYDATA:stack-trace})?"} } } else{ grok { patterns_dir => ["./patterns"] match => {"message" => "%{LINE_BEGIN:start-of-line} %{CUSTOM_TIMESTAMP:the-time} %{THREAD_AGENT:agent} %{USER:user-ID} %{REQUEST_ID:request-ID} %{STATUS:status} %{GREEDYDATA:message-body}"} }} date { match => ["timestamp", "dd/MM/yyyy HH:mm:ss.SSS"] } } output { stdout { codec => rubydebug } } ``` and the file that works has the following contents: ``` >>> 12/11/2015 15:50:08.054 [http-0.0.0.0-8080-3] [Anonymous@10.0.0.1, 10.0.0.1] [reqId:2611] ERROR - .integrations.ws.WSClient - Error occurred while calling [SystemLogon] WS >>> 11/11/2015 15:50:08.054 [http-0.0.0.0-8080-3] [Anonymous@10.0.0.1, 10.0.0.1] [reqId:2611] ERROR - s.ws.WSClient - Error occurred while calling [SystemLogon] java.lang.RuntimeException: java.lang.RuntimeException: Error occurred while making a SOAP call to [https://quality-] at EIDWSClient.makeSOAPcall(EIDWSClient.java:59) at call(SystemLogonWSClient.java:74) ``` Logstash is able to parse and grok (and everything) the above file without problems However if I transfer the first line (\>\>\> 12/11/2015...) after this line "at call(SystemLogonWSClient.java:74)" Logstash just hangs, unable to parse it. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 5, 2016, 2:12am UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255/2 "2016-04-05T02:12:14Z")
It "hangs" because it is expecting that to be a multiline event and only ever sees the start of that event, not the end, so it cannot move to the next event. ---
**Author:** ![pantheo](https://avatars.discourse-cdn.com/v4/letter/p/9de0a6/32.png) [@pantheo](https://discuss.elastic.co/u/pantheo)\ **Post date:** [April 5, 2016, 5:16am UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255/3 "2016-04-05T05:16:54Z")
Thanks for your reply. I was thinking the same thing but can not figure out why it cannot move to the next event. Shouldn't it just break out of the multiline event when the pattern is not satisfied? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 5, 2016, 5:26am UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255/4 "2016-04-05T05:26:07Z")
No, that'd defeat the purpose. ---
**Author:** ![pantheo](https://avatars.discourse-cdn.com/v4/letter/p/9de0a6/32.png) [@pantheo](https://discuss.elastic.co/u/pantheo)\ **Post date:** [April 5, 2016, 5:28am UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255/5 "2016-04-05T05:28:07Z")
So, I have to explicitly tell it when to break out? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 5, 2016, 5:28am UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255/6 "2016-04-05T05:28:48Z")
That is correct. ---
**Author:** ![pantheo](https://avatars.discourse-cdn.com/v4/letter/p/9de0a6/32.png) [@pantheo](https://discuss.elastic.co/u/pantheo)\ **Post date:** [April 5, 2016, 5:29am UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255/7 "2016-04-05T05:29:36Z")
Thank you for your help ---
**Author:** ![pantheo](https://avatars.discourse-cdn.com/v4/letter/p/9de0a6/32.png) [@pantheo](https://discuss.elastic.co/u/pantheo)\ **Post date:** [April 5, 2016, 5:43am UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255/8 "2016-04-05T05:43:03Z")
Just a last update for anyone that might have a similar problem. My problem is (exactly )what driskell refers to in his comment here: > > > We have events stored on the file system where an XML file represents 1 event. … These files are created in such a way that every file represents a new event. In each file, we don't have a newline indicator on the last line of the event. Following is a sample of an event where the last line does not have a new line indicator. Its basically the end of file. > > \`\`\` > \ xmlns:soap="http://www.w3.org/2001/12/soap-envelope" > soap:encodingStyle="http://www.w3.org/2001/12/soap-encoding"\> > > \ > \ > \34.5\ > \ > \ > > \ > \`\`\` > > Following is how my configuration looks like: > > \`\`\` > { > "general": { > "log level" : "debug" > }, > "network": { > "servers": \["127.0.0.1:5043" \], > "ssl ca": "/log-courier/bin/selfsigned.crt" > }, > "files": \[ > { > "paths": \["/logstash/logs/\*.xml" \], > "fields": { "type": "soap\_response" }, > "codec": { > "name": "multiline", > "pattern": "\<\\/.\*:Envelope\>", > "negate": true, > "what": "next" > } > } > \] > } > \`\`\` > > In the above case, the harvester keeps waiting for a new line character and will never return the event. Is there any configuration that can be done to return the event when the harvester hits the end of file? > Files are currently treated as streams of lines that are constantly > being written. So if we hit a line without EOL (end of line, \n) we > wait. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:03am UTC](https://discuss.elastic.co/t/grok-filter-cannot-parse-log-lines-that-are-after-a-multiline-event/46255/9 "2017-07-06T05:03:57Z")
Closed on July 6, 2017, 5:03am UTC