# Logstash reading new lines issues

**URL:** <https://discuss.elastic.co/t/logstash-reading-new-lines-issues/52163>\
**Category:** Logstash\
**Created:** [June 8, 2016, 7:07am UTC](https://discuss.elastic.co/t/logstash-reading-new-lines-issues/52163 "2016-06-08T07:07:12Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kennedy\_Kan1](https://avatars.discourse-cdn.com/v4/letter/k/cc9497/32.png) [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\
**Post date:** [June 8, 2016, 7:07am UTC](https://discuss.elastic.co/t/logstash-reading-new-lines-issues/52163/1 "2016-06-08T07:07:12Z")

</div>

I have written a conf file in logstash to read the jms log and the problems are I could not break the records into newlines fore filtering. Here is the raw data

```
####<Sep 20, 2015 12:00:12 AM> <> <1442678412960> <809000> <ID:<307061.1442678412716.0>> <> <CmsCorpAlsPrd_als_mod!CmsCorpAlsPrd_jmsAls_cdceap7e_32040@abb_audit_als_dQue> <Consumed> <<anonymous>> <MC:CA(local):OAMI(CmsCorpAlsPrd_cdceap7e_32040.jms.connection36.session121.consumer125)> <&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"&gt;&lt;mes:Header&gt;&lt;mes:JMSDeliveryMode&gt;PERSISTENT&lt;/mes:JMSDeliveryMode&gt;&lt;mes:JMSExpiration&gt;0&lt;> <> ####<Sep 20, 2015 12:00:13 AM> <> <1442678413018> <392000> <ID:<307061.1442678412943.0>> <> <CmsCorpAlsPrd_als_mod!CmsCorpAlsPrd_jmsAls_cdceap7e_32040@abb_audit_als_dQue> <Produced> <<anonymous>> <> <&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"&gt;&lt;mes:Header&gt;&lt;mes:JMSDeliveryMode&gt;PERSISTENT&lt;/mes:JMSDeliveryMode&gt;&lt;mes:JMSExpiration&gt;0&lt;> <> 

```

And here is my conf file in logstash

```
input{
	stdin{}
	file{
		type => "txt"
		path => "C:\HA\jms\jms.log"
		start_position => "beginning"
	}
}
filter{
	multiline{
		pattern => "\&"
		what => previous
	}	
	grok{
	match => {"message" => ['####<%{GREEDYDATA:Date}>%{SPACE}<>%{SPACE}<%{GREEDYDATA:Millisec_Date}>%{SPACE}<%{GREEDYDATA:Nanosec_Date}>%{SPACE}<ID:<%{GREEDYDATA:JMS_message_ID}>>%{SPACE}<>%{SPACE}<%{GREEDYDATA:JMS_destination_name}>%{SPACE}<%{GREEDYDATA:JMS_message_eventname}>%{SPACE}<<%{GREEDYDATA:JMS_username}>>%{SPACE}<%{GREEDYDATA:JMS_correlationID}>%{SPACE}<%{GREEDYDATA:Mcls}:JMSDeliveryMode&gt;%{WORD:JMSDeliveryMode}&lt;/mes:JMSDeliveryMode&gt;&lt;mes:JMSExpiration&gt;%{NUMBER:JMSExpiration}&lt;>%{SPACE}<>']}
    	}
}
output{
  	elasticsearch { hosts => ["localhost:9200"] 
	}
  	stdout { codec => rubydebug }	
}

```

All the things went well except when I run the conf and the result gives me this

```
                 "@version" => "1",
               "@timestamp" => "2016-06-08T06:23:50.543Z",
                     "path" => "C:\\HA\\jms\\jms.log",
                     "host" => "WIN-07LLQEN2SJB",
                     "type" => "txt",
                     "tags" => [
        [0] "multiline"
    ],
                     "Date" => "Sep 20, 2015 12:00:12 AM> <> <1442678412960> <809000> <ID:<307061.1
442678412716.0>> <> <CmsCorpAlsPrd_als_mod!CmsCorpAlsPrd_jmsAls_cdceap7e_32040@abb_audit_als_dQue> <Consumed> <<anonymou
s>> <MC:CA(local):OAMI(CmsCorpAlsPrd_cdceap7e_32040.jms.connection36.session121.consumer125)> <&lt;?xml version=\"1.0\"
encoding=\"UTF-8\"?&gt;\n&lt;mes:WLJMSMessage xmlns:mes=\"http://www.bea.com/WLS/JMS/Message\"&gt;&lt;mes:Header&gt;&lt;
mes:JMSDeliveryMode&gt;PERSISTENT&lt;/mes:JMSDeliveryMode&gt;&lt;mes:JMSExpiration&gt;0&lt;> <> \n####<
Sep 20, 2015 12:00:13 AM",
            "Millisec_Date" => "1442678413018",
             "Nanosec_Date" => "392000",
           "JMS_message_ID" => "307061.1442678412943.0",
     "JMS_destination_name" => "CmsCorpAlsPrd_als_mod!CmsCorpAlsPrd_jmsAls_cdceap7e_32040@abb_audit_als_dQue",
    "JMS_message_eventname" => "Produced",
             "JMS_username" => "anonymous",
                     "Mcls" => "&lt;?xml version=\"1.0\" encoding=\"UTF-8\"?&gt;\n&lt;mes:WLJMSMessage xmlns:mes=\"http:
//www.bea.com/WLS/JMS/Message\"&gt;&lt;mes:Header&gt;&lt;mes",
          "JMSDeliveryMode" => "PERSISTENT",
            "JMSExpiration" => "0"
}

```

Obviously, the date has part has read all the data in first message and seem classified i as the data of the 2nd message. Is there anyway to solve this in breaking different records in new line?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 8, 2016, 7:11am UTC](https://discuss.elastic.co/t/logstash-reading-new-lines-issues/52163/2 "2016-06-08T07:11:03Z")

</div>

Two things that I think will help:

- Don't use GREEDYDATA to match the date. As its name implies it's greedy. Use a more specific grok expression.
- A better multiline condition would probably be " **unless** the line begins with `###<` join with the previous line". If you insist on sticking with your current pattern you should at least prepend it with `^` so that it only matches ampersands at the beginning of the line.

---

<div class="post-metadata">

**Author:** ![Kennedy\_Kan1](https://avatars.discourse-cdn.com/v4/letter/k/cc9497/32.png) [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\
**Post date:** [June 8, 2016, 7:46am UTC](https://discuss.elastic.co/t/logstash-reading-new-lines-issues/52163/3 "2016-06-08T07:46:33Z")

</div>

Yes Thanks. Adding ^ helps a lot.  
However, when I run the conf file, it also gives me the **map parsing error status:400** , I would know what will be the issue for that?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 8, 2016, 6:01pm UTC](https://discuss.elastic.co/t/logstash-reading-new-lines-issues/52163/4 "2016-06-08T18:01:25Z")

</div>

Please show the full error message.

---

<div class="post-metadata">

**Author:** ![Kennedy\_Kan1](https://avatars.discourse-cdn.com/v4/letter/k/cc9497/32.png) [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\
**Post date:** [June 9, 2016, 5:34am UTC](https://discuss.elastic.co/t/logstash-reading-new-lines-issues/52163/5 "2016-06-09T05:34:22Z")

</div>

1. I turn out find out the sollution here: [github.com/elastic/elasticsearch/issues/16283](http://github.com/elastic/elasticsearch/issues/16283)
2. Another problem is the created field for indexing is too long. Shortening the name can solve the issue.

But I have no idea why `template_overwrite => true` can solve the issues

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:53am UTC](https://discuss.elastic.co/t/logstash-reading-new-lines-issues/52163/6 "2017-07-06T04:53:57Z")

</div>