# Multiline not picking up events immediately

**URL:** https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823
**Category:** Logstash
**Created:** [April 8, 2016, 3:37pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823 "2016-04-08T15:37:18Z")
**Posts on this page:** 18
**Page:** 1

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 8, 2016, 3:37pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/1 "2016-04-08T15:37:18Z")

</div>

This is my conf file -

file{  
path =\> "C:/Logs/\*"  
codec =\> multiline  
{  
pattern =\> "%{WORD:EventId} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:EventLevel}, %{WORD:Message} : %{WORD:Operation}, %{WORD:Payload} : [%{WORD:Session\_ID} : %{UUID:SessionID}] [%{WORD:IPAddress} : %{IPORHOST:ClientIPAddress}] [%{WORD:Method} : %{WORD:HTTP\_Verb}] [%{WORD:requestUri} : %{URI:URL}] [%{WORD:content} : %{DATA:Request}] , %{WORD:EventName} : %{WORD:EventNameValue}, %{WORD:Timestamp} : %{TIMESTAMP\_ISO8601:Request\_DateTime}, %{WORD:ProcessId} : %{NUMBER:ProcessID}, %{WORD:ThreadID} : %{NUMBER:ThreadID}"  
what =\> next  
}  
}

The problem is - Multiline not picking up the events as soon as the log is written. It is picking up the the first one only when the second one with the same pattern is written.

eg. my first log -  
EventId : 1, Level : Informational, Message : Request, Payload : [sessionID : 3e4ad1b0-2d2d-4f77-90b3-6315aacfacb7] [ipAddress : 67.29.26.92] [method : GET] [requestUri : [http://server1:8080/MyService/api/products/6900/availability/](http://server1:8080/MyService/api/products/6900/availability/)] [content :] , EventName : RequestInfo, Timestamp : 2016-04-05T14:35:36.4947170Z, ProcessId : 8924, ThreadId : 3008

my second log -  
EventId : 1, Level : Informational, Message : Request, Payload : [sessionID : 3e4ad1b0-2d2d-4f77-90b3-6315aacfacb7] [ipAddress : 67.29.26.92] [method : GET] [requestUri : [http://server1:8080/MyService/api/products/6900/availability/](http://server1:8080/MyService/api/products/6900/availability/)] [content :] , EventName : RequestInfo, Timestamp : 2016-04-05T14:35:36.4947170Z, ProcessId : 8924, ThreadId : 3008

My first log is captured only when the my second log is written. I tried both "what =\> next and what =\> previous". Same result.

What am i missing here ??

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [April 9, 2016, 7:12am UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/2 "2016-04-09T07:12:13Z")

</div>

Because it needs to wait for the next event to know when the previous one has finished.

---

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 9, 2016, 3:00pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/3 "2016-04-09T15:00:36Z")

</div>

But that would be a problem for me. Because in my case, I send a request to my service and i get a response back.

If i need to see the values in Kibana, logstash needs to pull that up immediately.

But now, I have to fire another request-response, then the first one will be captured. Don't you think this is wierd or is there anything that I am missing here.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [April 10, 2016, 8:31pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/4 "2016-04-10T20:31:36Z")

</div>

> pattern =\> "%{WORD:EventId} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:EventLevel}, %{WORD:Message} : %{WORD:Operation}, %{WORD:Payload} : [%{WORD:Session\_ID} : %{UUID:SessionID}] [%{WORD:IPAddress} : %{IPORHOST:ClientIPAddress}] [%{WORD:Method} : %{WORD:HTTP\_Verb}] [%{WORD:requestUri} : %{URI:URL}] [%{WORD:content} : %{DATA:Request}] , %{WORD:EventName} : %{WORD:EventNameValue}, %{WORD:Timestamp} : %{TIMESTAMP\_ISO8601:Request\_DateTime}, %{WORD:ProcessId} : %{NUMBER:ProcessID}, %{WORD:ThreadID} : %{NUMBER:ThreadID}"

This is functionally equivalent to:

```
pattern => "%{WORD} : %{NUMBER}, %{WORD} : ...

```

But again, use literal strings when you want literal matches:

```
pattern => "EventId : %{NUMBER}, Level : ...

```

Where's the newline character in these log examples, i.e. which two (or more) lines do you want to join? Your multiline pattern doesn't make sense to me and since I don't understand what your input looks like I can't make an improvement suggestion.

---

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 10, 2016, 10:59pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/5 "2016-04-10T22:59:46Z")

</div>

I got this to be working. In my above example, multiline was not present. Here is another example where I need to have multiline func.  
I have reduced my log example here -

Logs is in below format and keeps on writing on basis of my service hits. Every service call below 2 lines are written.

Log 1 -  
ProviderId : 46875d78-3924-5303-f49e-65b8bc0dc289  
EventId : 1

Log 2 -  
ProviderId : 46875d78-3924-5303-f49e-65b8bc0dc290  
EventId : 2

Log 1 is not captured until log 2 is written. Thats my prb. I wanted log 1 to be captured as soon as it is wrritten in the log file.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [April 11, 2016, 5:44am UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/6 "2016-04-11T05:44:34Z")

</div>

It seems a configuration like

```auto
pattern => "^ProviderId: "
what => "next"

```

should work and give the desired result. Your most recent example doesn't match the configuration you posted earlier to it's hard to tell what's wrong.

---

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 11, 2016, 5:59am UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/7 "2016-04-11T05:59:20Z")

</div>

You can ignore my first post.

I have my conf with (pattern =\> "^ProviderId: " what =\> "next"). It is capturing the event. The only prb is "it is not capturing the log as soon as it is written to my log file". It is captured only when the next log with same pattern is written into the same file. That is because it is looking for the pattern.

I have figured out a solution for this. I wont be using multiline. I will be using multiple match patterns.  
But now another prb has arised ☹ Need to capture, entire stack trace.

Payload : [sessionID : 20cb9749-3db7-4bb2-bd72-5f397664ed24] [type : InvalidMethodArgumentException] [data :] [innerException :] [message : The CallID field is required.] [source : MySol.Services.Common] [stackTrace : at MySol.Services.Common.Entities.Database.BaseEFService`1.<SaveAsync>d__0.MoveNext() in e:\Builds\15\CR\Main.QA\src\MyWebService\MySol.Services.Common\Entities\Database\BaseEFService.cs:line 101 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at MySol.Services.Common.Entities.Database.BaseEFCreateService`3.d\_\_3.MoveNext() in e:\Builds\15\CR\[Main.QA](http://Main.QA)\src\MyWebService\MySol.Services.Common\Entities\Database\BaseEFCreateService.cs:line 47  
--- End of stack trace from previous location where exception was thrown ---  
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)  
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
at MySol.API.Common.MessageHandlers.CreateCRRRequestHeadersMessageHandler.d\_\_0.MoveNext() in e:\Builds\15\CR\[Main.QA](http://Main.QA)\src\MyWebService\MySol.API.Common\MessageHandlers\CreateCRRRequestHeadersMessageHandler.cs:line 58] [targetSite : Void MoveNext()]

Got this working in GROK debugger - [stackTrace : (?(.|\r|\n)\*)]. But this is not working in my logstash.conf file.

What am I missing here magnus ?

---

<div class="post-metadata">

### Author: ![Suren92](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@Suren92](https://discuss.elastic.co/u/Suren92)
#### Post date: [April 11, 2016, 10:41am UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/8 "2016-04-11T10:41:18Z")

</div>

Can you post your config file here?

---

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 11, 2016, 2:50pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/9 "2016-04-11T14:50:36Z")

</div>

Conf -

input  
{  
file  
{  
path =\> "C:/MyLogs/\*"  
#start\_position =\> "beginning"  
}  
}

filter  
{  
grok  
{  
match =\>  
{  
"message" =\>  
[  
"ProviderId : %{UUID:SessionID}",  
"EventId : %{NUMBER:EventId}",  
"Payload : [sessionID : %{UUID:SessionID}] [type : %{WORD:ExceptionType}] [data : %{DATA:Data}] [innerException : %{DATA:InnerException}] [message : %{DATA:ExceptionMessage}] [source : %{DATA:ExceptionSource}] [stackTrace : (?(.|\r|\n)\*)]",  
"EventName : %{WORD:EventName}",  
"ProcessId : %{NUMBER:ProcessId}",  
"ThreadId : %{NUMBER:ThreadId}"  
]  
}  
}

```
mutate
{
	add_field => { "Type" => "ApiExceptionEventSource" }
	
	rename => { "path" => "Path" }
	rename => { "host" => "Host" }
}

```

}

output  
{  
stdout  
{  
codec =\> rubydebug  
}  
}

Log Pattern -

ProviderId : 46875d78-3924-5303-f49e-65b8bc0dc289  
EventId : 11  
Payload : [sessionID : 19c6fd3f-79d5-406a-9d93-f1ada6bc6fde] [type : InvalidMethodArgumentException] [data :] [innerException : ExceptionMessage] [message : The CallID field is required.] [source : MySol.Services.Common] [stackTrace : at MySol.Services.Common.Entities.Database.BaseEFService`1.<SaveAsync>d__0.MoveNext() in e:\Builds\15\IR\Main.QA\src\WebServiceSln\MySol.Services.Common\Entities\Database\BaseEFService.cs:line 101 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at MySol.Services.Common.Entities.Database.BaseEFCreateService`3.d\_\_3.MoveNext() in e:\Builds\15\IR\[Main.QA](http://Main.QA)\src\WebServiceSln\MySol.Services.Common\Entities\Database\BaseEFCreateService.cs:line 47  
--- End of stack trace from previous location where exception was thrown ---  
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)  
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
at MySol.API.Common.MessageHandlers.CreateIRRRequestHeadersMessageHandler.d\_\_0.MoveNext() in e:\Builds\15\IR\[Main.QA](http://Main.QA)\src\WebServiceSln\MySol.API.Common\MessageHandlers\CreateIRRRequestHeadersMessageHandler.cs:line 58] [targetSite : Void MoveNext()]  
EventName : ResponseInfo  
ProcessId : 21096  
ThreadId : 12336

---

<div class="post-metadata">

### Author: ![Alex\_6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_6/32/1330_2.png) [@Alex\_6](https://discuss.elastic.co/u/Alex_6)
#### Post date: [April 11, 2016, 3:55pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/10 "2016-04-11T15:55:14Z")

</div>

Doesn't the [`max_age`](https://www.elastic.co/guide/en/logstash/current/plugins-filters-multiline.html#plugins-filters-multiline-max_age) parameter work? By default a multiline message should be flushed after waiting 5 seconds for additional lines. What version of Logstash are you using?

---

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 11, 2016, 4:13pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/11 "2016-04-11T16:13:19Z")

</div>

I havent tried the max\_age parameter. Will look into that. I am using the logstash 2.2.2 version.

---

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 12, 2016, 5:46am UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/12 "2016-04-12T05:46:39Z")

</div>

The max\_age worked.

multiline  
{  
pattern =\> "^ProviderId : "  
what =\> "previous"  
negate =\> true  
max\_age =\> 3  
}

Also, i saw the same issue logged on github "[https://github.com/logstash-plugins/logstash-codec-multiline/issues/11](https://github.com/logstash-plugins/logstash-codec-multiline/issues/11)". The fix was the same.

There is only one thing concerning me here is about the THREAD. What challenges/issues will i face being on single thread ?

←[33mDefaulting pipeline worker threads to 1 because there are some filters that  
might not work with multiple worker threads {:count\_was=\>4, :filters=\>["multili  
ne"], :level=\>:warn}←[0m

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [April 12, 2016, 5:54am UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/13 "2016-04-12T05:54:24Z")

</div>

Running just one pipeline worker could obviously impact the total throughput.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 12, 2016, 6:04am UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/14 "2016-04-12T06:04:00Z")

</div>

It is also worth noting that [the multiline filter is being deprecated](https://www.elastic.co/blog/logstash-moving-away-from-node-protocol-and-multiline-filter). A solution might be to use Fiebeat to send data to Logstash, as it [supports multiline processing](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#multiline) and allows a timeout to be configured to ensure the last event is sent.

---

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 12, 2016, 1:25pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/15 "2016-04-12T13:25:06Z")

</div>

Thanks Christian. I was using FB prev. But removed it since log stash was doing everything it did.

As per your suggestion, i will defn look into the link and will use fb for multiline processing.  
If you could provide an sample multiline in fb link, it would be more helpful.

---

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 12, 2016, 11:22pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/16 "2016-04-12T23:22:23Z")

</div>

I tried using the multiline in FB. It worked. But, when i tried sending the FB data to Logstash, i could see duplicate entries happening for each log captured.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/c/cbe482a8086181c980da7357f26133b06ba00a29.png)

LS Conf -  
input { beats { port =\> 4434 } }  
output { stdout { codec =\> rubydebug } }

FB -  
multiline:  
pattern: "^ProviderId :"  
negate: true  
match: after

What am I missing here ? Why is it happening twice ?

---

<div class="post-metadata">

### Author: ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)
#### Post date: [April 13, 2016, 12:44pm UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/17 "2016-04-13T12:44:03Z")

</div>

Any update on this please ?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:02am UTC](https://discuss.elastic.co/t/multiline-not-picking-up-events-immediately/46823/18 "2017-07-06T05:02:28Z")

</div>