# Yet another multiline plea for help please **URL:** \ **Category:** Logstash\ **Created:** [November 9, 2016, 6:08pm UTC](https://discuss.elastic.co/t/yet-another-multiline-plea-for-help-please/65536 "2016-11-09T18:08:50Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ZillaG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zillag/32/10505_2.png) [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Post date:** [November 9, 2016, 6:08pm UTC](https://discuss.elastic.co/t/yet-another-multiline-plea-for-help-please/65536/1 "2016-11-09T18:08:50Z")
I've looked over multiple posts including [this one](https://discuss.elastic.co/t/multiline-problem/26219) on how to parse multiline log patterns like I have below. Seems like I'm following the solutions given, but Kibana still shows my logs into multiple documents. Here's my Logstash conf file input plugin ``` input { tcp { port => 1234 codec => json codec => multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" } } } ``` ... and here's a sample log ``` 2016-10-31 19:01:21,144 ERROR :localhost-startStop-1 [com.teradata.trm.common.workflow.AbstractTask] Error caught in task execution com.company.trm.common.workflow.TaskException: 10.110 Executable command $TMI_COMMAND_MANAGER_PATH$ -n trm_project_source -u Administrator -p -f "/usr/share/tomcat8/dist/trm/tmp_cmdmgr_script_175829326783761307893.scp" -o "/usr/share/tomcat8/dist/trm/logs/commandManager.log" completed with error. Return code = 127. at com.teradata.trm.pe.common.task.ProcessExecutorTask.handleReturnCode(ProcessExecutorTask.java:156) at com.teradata.trm.reports.CommandManagerExecutorTask.handleReturnCode(TRMCommandManagerImpl.java:295) at com.teradata.trm.pe.common.task.ProcessExecutorTask.doExecute(ProcessExecutorTask.java:79) at com.teradata.trm.common.workflow.AbstractTask.execute(AbstractTask.java:492) at com.teradata.trm.reports.TRMCommandManagerImpl.executeCommandManager(TRMCommandManagerImpl.java:141) at com.teradata.trm.reports.TRMCommandManagerImpl.updateReportsDataWareHouse(TRMCommandManagerImpl.java:53) at com.teradata.trm.reports.ReportsServiceImpl.checkAndSyncReportsWarehouseDatabaseInfo(ReportsServiceImpl.java:199) at com.teradata.trm.reports.ReportsServiceImpl.syncTRMAndReports(ReportsServiceImpl.java:68) at com.teradata.trm.common.reports.ReportsInitializer.initialize(ReportsInitializer.java:29) at com.teradata.trm.common.context.Initializer.onApplicationEvent(Initializer.java:73) at com.teradata.trm.common.context.Initializer.onApplicationEvent(Initializer.java:30) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:324) at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:929) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:467) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:385) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:284) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4729) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5167) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:945) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1768) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.Exception at com.teradata.trm.common.workflow.TaskException.(TaskException.java:94) ... 31 more ``` Any advice? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 10, 2016, 6:44am UTC](https://discuss.elastic.co/t/yet-another-multiline-plea-for-help-please/65536/2 "2016-11-10T06:44:40Z")
You can't have two `codec` options and I'm not sure what happens if you do. Remove `codec => json` and try again. (Since the data isn't JSON it didn't make any sense to have it here in the first place.) Using the multiline codec on a TCP stream seems fragile. I would avoid it. ---
**Author:** ![ABecker](https://avatars.discourse-cdn.com/v4/letter/a/ebca7d/32.png) [@ABecker](https://discuss.elastic.co/u/ABecker)\ **Post date:** [November 10, 2016, 7:39am UTC](https://discuss.elastic.co/t/yet-another-multiline-plea-for-help-please/65536/3 "2016-11-10T07:39:40Z")
Hello ZillaG, I think the problem within your configuration is that you have to decide which codec you want to use, multiline or json. If I look on your log sample I would say that json is wrong is definitive wrong. Your configuration should look like input { tcp { ``` port => 1234 codec => multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" } ``` } } If you still have problems that your entries will be cut, it should be necessary to extend your max\_lines limit. The defalt is set to 500 lines I think. You can extend it by using the following configuration. max\_lines =\> 1500 But be aware. It cost a more and more performance if you increase your max\_lines. I hope this answer your question. Mit freundlichen Grüßen/Best regards Artur Becker METRO SYSTEMS GmbH System Management Platforms Contact Address: METRO SYSTEMS GmbH Metro-Strasse 12, 40235 Duesseldorf, Germany Phone: +49 (211) 969-9307 E-Mail: [artur.becker@metrosystems.net](mailto:artur.becker@metrosystems.net)[mailto:artur.becker@metrosystems.net](mailto:artur.becker@metrosystems.net) Internet: [www.metrosystems.net](http://www.metrosystems.net)[http://www.metrosystems.net](http://www.metrosystems.net) ü SAVE PAPER - THINK BEFORE YOU PRINT ---
**Author:** ![ZillaG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zillag/32/10505_2.png) [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Post date:** [November 10, 2016, 1:44pm UTC](https://discuss.elastic.co/t/yet-another-multiline-plea-for-help-please/65536/4 "2016-11-10T13:44:33Z")
Thank you both @magnusbaeck and @ABecker. My dilemma is I send my logs via rsyslog from a remote servers, and there I'm using an rsyslog template that packs the messages into json. So if I remove the codec =\> json, I don't see the messages handled by logstash. Here's my rsyslog template on the remote host ``` template(name="textLogTemplate" type="list") { constant(value="{ ") constant(value="\"type\":\"") property(name="programname") constant(value="\", ") constant(value="\"host\":\"") property(name="hostname") constant(value="\", ") constant(value="\"timestamp\":\"") property(name="timestamp" dateFormat="rfc3339") constant(value="\", ") constant(value="\"@version\":\"1\", ") constant(value="\"customer\":\"my_customer\", ") constant(value="\"role\":\"my_app_server\", ") constant(value="\"sourcefile\":\"") property(name="$!metadata!filename") constant(value="\", ") constant(value="\"message\":\"") property(name="rawmsg" format="json") constant(value="\"}\n") } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 10, 2016, 1:51pm UTC](https://discuss.elastic.co/t/yet-another-multiline-plea-for-help-please/65536/5 "2016-11-10T13:51:54Z")
Why do you need the multiline codec if each message is serialized as JSON? Pleas show us _exactly_ what a payload looks like. You can capture it with e.g. netcat or Wireshark. ---
**Author:** ![ZillaG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zillag/32/10505_2.png) [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Post date:** [November 10, 2016, 2:38pm UTC](https://discuss.elastic.co/t/yet-another-multiline-plea-for-help-please/65536/6 "2016-11-10T14:38:06Z")
@magnusbaeck, yes I think now my problem in the source that is, the client that's sending the logs. I may not be serializing my data correctly, so I'll look into that. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2016, 2:38pm UTC](https://discuss.elastic.co/t/yet-another-multiline-plea-for-help-please/65536/7 "2016-12-08T14:38:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.