# Issue with Multiline Logs

**URL:** <https://discuss.elastic.co/t/issue-with-multiline-logs/97432>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 17, 2017, 1:53pm UTC](https://discuss.elastic.co/t/issue-with-multiline-logs/97432 "2017-08-17T13:53:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vivek\_Sharma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_sharma/32/46948_2.png) [@Vivek\_Sharma](https://discuss.elastic.co/u/Vivek_Sharma)\
**Post date:** [August 17, 2017, 1:53pm UTC](https://discuss.elastic.co/t/issue-with-multiline-logs/97432/1 "2017-08-17T13:53:24Z")

</div>

Hi

I am trying to read log file with multilines for a single log.

I have configured filebeat.yml as below:

> ```
> > - input_type: log 
> > paths:
> > - C:\ELK\*.log
> > fields: 
> > tech_stack: XXX
> > kpi_type: YYY
> > 
> > multiline.pattern: '^([Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec])\w+\s(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\s(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)'
> > multiline.negate: true
> > 
> > multiline.match: after
> 
> ```

Logstash has been defined as below:

```
> input {
> beats {
> port => 5044
> }
>    
> }
> 
> filter {
> grok {
> match => { "message" => [
> #	For reading STO log files. This will be common for both start and stop service success scenario
> "%{SYSLOGTIMESTAMP:timestamp} \[Host:%{HOSTNAME:host_name}\:-1\|Service:%{PROG:log_service_name}\|Context:%{USERNAME:context}\|Session:%{USERNAME:session_id}\|User:%{USERNAME:user_name}\|ApplicationId:%{USERNAME:application_id}\|MessageId:%{USERNAME:message_id}\|CorrelationId:%{USERNAME:correlation_id}\|GeneratingSystem:%{USERNAME:generating_system}\|LogMessage:%{GREEDYDATA:log_message}\|MessageBody:STO_NUMBER:%{USERNAME:sto_number}]\|"
> ] }
> }
> 	
> mutate {
> add_field => { "tech_stack" => "%{[fields][tech_stack]}" }
> add_field => { "kpi_type" => "%{[fields][kpi_type]}" }
> }
> }
> 
> output {
> elasticsearch {
> hosts => "localhost:9200"
> user => "elastic"
> password => "changeme"
> manage_template => false
> index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
> document_type => "%{[@metadata][type]}" 
> }
> stdout { codec => rubydebug }
> }

```

Now my logs are as below:

> Jul 29 19:16:43 [Host:dlap-w1is0419.xxxxxxxxx.com:-1|Service:RPU\_WWW\_RT\_CC\_Utils.LoggingFramework:logMessage|  
> Context:a9604140-f805-1469-9704-fffffffaa84d|Session:null|User:Administrator|ApplicationId:RT\_IA1424|MessageId:null|CorrelationId:d4e01dbb-17f8-4851-b54b-31074a4a57fb|GeneratingSystem:webMethods|  
> LogMessage:Service Start :RPU\_WWW\_RT\_PRP\_OrderDetails.Maps:mapXXXXXXXXXXXSvc|MessageBody:XXX\_NUMBER:1245789654]|

As per filebeat configuration, these three lines of log need to be merged into a single line and then passed to logstash. But logstash is receiving all these logs in separate lines.

Could anyone please help.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 17, 2017, 10:03pm UTC](https://discuss.elastic.co/t/issue-with-multiline-logs/97432/2 "2017-08-17T22:03:10Z")

</div>

Multiline support is based on regular expression. As the `[<chars>]` term creates a custom character class, I don't think `[Jan|Feb|...]` is doing what you'd expect. Use `(...)` to create an 'or' group. Like `(Jan|Feb|...)`. You can try to shorten the regex to this (the expression should capture a structural pattern, no need to be too strict on actual content): `'^[JFMASOND][a-z]{2} \d{2}:\d{2}:\d{2} \['`

Btw. since filebeat 5.3 (I think) you can use `()` to build groups. The matcher in libbeat applies some simple optimizations to the pattern. Like internally optimizing `(<terms>)` to `(?:<terms>)`.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2017, 10:03pm UTC](https://discuss.elastic.co/t/issue-with-multiline-logs/97432/3 "2017-09-14T22:03:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.