# Logstash adds extra new line characters **URL:** \ **Category:** Logstash\ **Created:** [September 16, 2016, 3:04am UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660 "2016-09-16T03:04:40Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![andreys](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@andreys](https://discuss.elastic.co/u/andreys)\ **Post date:** [September 16, 2016, 3:04am UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660/1 "2016-09-16T03:04:40Z")
I am using next configuration file ``` input { stdin { codec => multiline { pattern => "%{APEX_TIMESTAMP}" negate => true what => "previous" } } } filter { grok { patterns_dir => ["/etc/logstash/patterns"] match => ["message", "%{APEX_TIMESTAMP:timestamp} %{LOGLEVEL:severity} %{GREEDYDATA:message}"] } date { timezone => "US/Pacific" match => ["timestamp" , "yyyy-MM-dd HH:mm:ss,SSS"] } } output { elasticsearch { hosts => ["localhost"] } } ``` where `APEX_TIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND},\d{3}` as result I got 13 log messages from ~41,000 with **\_grokparsefailure**. This is one of them ``` { "_index": "logstash-2016.09.16", "_type": "logs", "_id": "AVcw6Si2vjCqs-IM_UH9", "_score": null, "_source": { "@timestamp": "2016-09-16T02:52:49.123Z", "message": "2016-07-27 15:46:30,145 D\nEBUG [defaultQuartzScheduler_Worker-21] [XpsMessageContext:204] ( ) [??????????] createXpsMessageTasks: { Message Task for UPDATE, table ID(34), entity(com.????????????), message key(34:70582107), entity gkey (?????????), mediator (class com.????????)}", "@version": "1", "tags": [ "multiline", "_grokparsefailure" ], "host": "localhost.localdomain" ``` but the original message from log file does not have "\n" in DEBUG ``` [root@localhost Backup]# egrep '^2016-07-27 15:46:30,145 .*message key\(34:70582107\).*' navis-apex.log |od -c 0000000 2 0 1 6 - 0 7 - 2 7 1 5 : 4 6 0000020 : 3 0 , 1 4 5 D E B U G [ 0000040 d e f a u l t Q u a r t z S c h 0000060 e d u l e r _ W o r k e r - 2 1 ``` Where did I make configuration mistake? Thank you ---
**Author:** ![Alex\_6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_6/32/1330_2.png) [@Alex\_6](https://discuss.elastic.co/u/Alex_6)\ **Post date:** [September 16, 2016, 10:53am UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660/2 "2016-09-16T10:53:05Z")
Hi, can you post a normal plain text example from your log? Also in your output just stick in the following instead of the Elasticsearch output for now and see if Logstash is fine: ``` stdout { codec => rubydebug } ``` ---
**Author:** ![andreys](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@andreys](https://discuss.elastic.co/u/andreys)\ **Post date:** [September 16, 2016, 5:48pm UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660/3 "2016-09-16T17:48:44Z")
Sure. ``` [root@localhost Backup]# /opt/logstash/bin/logstash -f /root/Backup/first-pipeline.conf < navis-apex.log > out [root@localhost Backup]# egrep '2016-07-27T22:46:30.145Z' --color -A 10 -B 10 out "@timestamp" => "2016-09-16T17:43:06.219Z", "message" => "2016-07-27 15:46:30,145 D\nEBUG [defaultQuartzScheduler_Worker-21] [XpsMessageContext:204] ( ) [DPW/DPWA/AGW/AGW] createXpsMessageTasks: { Message Task for UPDATE, table ID(34), entity(com.navis.inventory.business.units.UnitYardVisit), message key(34:70582107), entity gkey (70582107), mediator (class com.navis.apex.business.xps.mediators.XpmContainerArgo)}", "@version" => "1", "tags" => [ [0] "multiline", [1] "_grokparsefailure" ], "host" => "localhost.localdomain" } { "@timestamp" => "2016-07-27T22:46:30.145Z", "message" => " [defaultQuartzScheduler_Worker-21] [XpsMessageContext:204] ( ) [DPW/DPWA/AGW/AGW] createXpsMessageTasks: { Message Task for UPDATE, table ID(34), entity(com.navis.inventory.business.units.UnitYardVisit), message key(34:70582107), entity gkey (70582107), mediator (class com.navis.apex.business.xps.mediators.XpmContainerArgo)}", "@version" => "1", "host" => "localhost.localdomain", "timestamp" => "2016-07-27 15:46:30,145", "severity" => "DEBUG" } { "@timestamp" => "2016-07-27T22:46:30.153Z", "message" => " [ajp-bio-10009-exec-8] [GroovyApi:229] ( ) [DPWAntwerpUnitReroute] GATE_21204 : e=null", "@version" => "1", [root@localhost Backup]# egrep '2016-07-27 15:46:30,145 D' navis-apex.log 2016-07-27 15:46:30,145 DEBUG [defaultQuartzScheduler_Worker-21] [XpsMessageContext:204] ( ) [DPW/DPWA/AGW/AGW] createXpsMessageTasks: { Message Task for UPDATE, table ID(34), entity(com.navis.inventory.business.units.UnitYardVisit), message key(34:70582107), entity gkey (70582107), mediator (class com.navis.apex.business.xps.mediators.XpmContainerArgo)} 2016-07-27 15:46:30,145 DEBUG [defaultQuartzScheduler_Worker-21] [XpsMessageContext:204] ( ) [DPW/DPWA/AGW/AGW] createXpsMessageTasks: { Message Task for UPDATE, table ID(34), entity(com.navis.inventory.business.units.UnitYardVisit), message key(34:70582107), entity gkey (70582107), mediator (class com.navis.apex.business.xps.mediators.XpmContainerArgo)} ``` ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [September 19, 2016, 4:58pm UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660/4 "2016-09-19T16:58:44Z")
@andreys This is a known problem when the `multiline codec` is used with `stdin input`. The stdin input sends a 16K chunk of text to the multiline codec. The multiline codec splits the chunk by `new line characters` but last piece is usually not a full "line" but the multiline codec thinks it is and the rest of the original line (in the next 16K chunk) does not start with `APEX_TIMESTAMP` and therefore it gets appended to the `previous` lines with a new line. There is no real solution to this while still using those two plugins together. You should try to find an alternative input to use. ---
**Author:** ![andreys](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@andreys](https://discuss.elastic.co/u/andreys)\ **Post date:** [September 19, 2016, 6:42pm UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660/5 "2016-09-19T18:42:00Z")
@guyboertje Thank you! I installed Beats on remote servers and configure them to pass logs to logstash. Will check if I can find this kind of problems in this setup. With this bug, which you described, how can I push all old logs to ElasticSearch? I have at least terabyte of data which was collected for last several months. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [September 21, 2016, 11:47am UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660/6 "2016-09-21T11:47:05Z")
Use filebeat with the beats input in LS to read each file. ---
**Author:** ![jmasip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmasip/32/13210_2.png) [@jmasip](https://discuss.elastic.co/u/jmasip)\ **Post date:** [November 16, 2016, 4:47pm UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660/7 "2016-11-16T16:47:46Z")
@andreys For the Navis N4, just creating a new appender in N4 you can drop the logs into log4j without the log files. Unfortunately, for XPS (service and clients), you have to use the files. 😉 cheers. Jordi ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:30am UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660/8 "2017-07-06T04:30:29Z")