Yet another multiline plea for help please

Elastic Stack Logstash
1

I've looked over multiple posts including this one on how to parse multiline log patterns like I have below. Seems like I'm following the solutions given, but Kibana still shows my logs into multiple documents.

Here's my Logstash conf file input plugin

input {
  tcp {
    port => 1234
    codec => json
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
    }
  }
}

... and here's a sample log

2016-10-31 19:01:21,144 ERROR :localhost-startStop-1 [com.teradata.trm.common.workflow.AbstractTask]  Error caught in task execution
com.company.trm.common.workflow.TaskException: 10.110 Executable command $TMI_COMMAND_MANAGER_PATH$ -n trm_project_source -u Administrator -p <password> -f "/usr/share/tomcat8/dist/trm/tmp_cmdmgr_script_175829326783761307893.scp" -o "/usr/share/tomcat8/dist/trm/logs/commandManager.log" completed with error. Return code = 127.
        at com.teradata.trm.pe.common.task.ProcessExecutorTask.handleReturnCode(ProcessExecutorTask.java:156)
        at com.teradata.trm.reports.CommandManagerExecutorTask.handleReturnCode(TRMCommandManagerImpl.java:295)
        at com.teradata.trm.pe.common.task.ProcessExecutorTask.doExecute(ProcessExecutorTask.java:79)
        at com.teradata.trm.common.workflow.AbstractTask.execute(AbstractTask.java:492)
        at com.teradata.trm.reports.TRMCommandManagerImpl.executeCommandManager(TRMCommandManagerImpl.java:141)
        at com.teradata.trm.reports.TRMCommandManagerImpl.updateReportsDataWareHouse(TRMCommandManagerImpl.java:53)
        at com.teradata.trm.reports.ReportsServiceImpl.checkAndSyncReportsWarehouseDatabaseInfo(ReportsServiceImpl.java:199)
        at com.teradata.trm.reports.ReportsServiceImpl.syncTRMAndReports(ReportsServiceImpl.java:68)
        at com.teradata.trm.common.reports.ReportsInitializer.initialize(ReportsInitializer.java:29)
        at com.teradata.trm.common.context.Initializer.onApplicationEvent(Initializer.java:73)
        at com.teradata.trm.common.context.Initializer.onApplicationEvent(Initializer.java:30)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:324)
        at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:929)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:467)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:385)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:284)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4729)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5167)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:945)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1768)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception
        at com.teradata.trm.common.workflow.TaskException.<init>(TaskException.java:94)
        ... 31 more

Any advice?

2

You can't have two codec options and I'm not sure what happens if you do. Remove codec => json and try again. (Since the data isn't JSON it didn't make any sense to have it here in the first place.)

Using the multiline codec on a TCP stream seems fragile. I would avoid it.

3

Hello ZillaG,

I think the problem within your configuration is that you have to decide which codec you want to use, multiline or json.

If I look on your log sample I would say that json is wrong is definitive wrong.

Your configuration should look like

input {

tcp {

port => 1234

codec => multiline {

  pattern => "^%{TIMESTAMP_ISO8601}"

  negate => true

  what => "previous"

}

}

}

If you still have problems that your entries will be cut, it should be necessary to extend your max_lines limit. The defalt is set to 500 lines I think. You can extend it by using the following configuration.

max_lines => 1500

But be aware. It cost a more and more performance if you increase your max_lines.

I hope this answer your question.

Mit freundlichen Grüßen/Best regards

Artur Becker

METRO SYSTEMS GmbH
System Management Platforms

Contact Address:
METRO SYSTEMS GmbH
Metro-Strasse 12, 40235 Duesseldorf, Germany
Phone:

+49 (211) 969-9307

E-Mail:

artur.becker@metrosystems.netmailto:artur.becker@metrosystems.net

Internet:

www.metrosystems.nethttp://www.metrosystems.net

ü SAVE PAPER - THINK BEFORE YOU PRINT

4

Thank you both @magnusbaeck and @ABecker. My dilemma is I send my logs via rsyslog from a remote servers, and there I'm using an rsyslog template that packs the messages into json. So if I remove the codec => json, I don't see the messages handled by logstash. Here's my rsyslog template on the remote host

template(name="textLogTemplate" type="list") {
  constant(value="{ ")

  constant(value="\"type\":\"")
  property(name="programname")
  constant(value="\", ")

  constant(value="\"host\":\"")
  property(name="hostname")
  constant(value="\", ")

  constant(value="\"timestamp\":\"")
  property(name="timestamp" dateFormat="rfc3339")
  constant(value="\", ")

  constant(value="\"@version\":\"1\", ")

  constant(value="\"customer\":\"my_customer\", ")

  constant(value="\"role\":\"my_app_server\", ")

  constant(value="\"sourcefile\":\"")
  property(name="$!metadata!filename")
  constant(value="\", ")

  constant(value="\"message\":\"")
  property(name="rawmsg" format="json")
  constant(value="\"}\n")
}
5

Why do you need the multiline codec if each message is serialized as JSON? Pleas show us exactly what a payload looks like. You can capture it with e.g. netcat or Wireshark.

6

@magnusbaeck, yes I think now my problem in the source that is, the client that's sending the logs. I may not be serializing my data correctly, so I'll look into that. Thanks!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.