Multi line logs Javastack trace not showing up correctly

Elastic Stack Beats
1

The java stack trace logs are not coming in as multi-lines they are still coming in as separate lines

Filebeat Config

      - /var/***/catalina*
     - /var/***/*.log
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:![image|689x269](upload://1gTJ1sNVcJL77EQrk6NvggwAlyE.png) `

image
image1212×474 29.2 KB

2

Could you please correct the formatting using code tags? Also, please provide sample logs here.

3

Hello mtojek,

I have the same issue, here under my logs:
2019 May 08 08:27:37:283 GMT -0400 BW.ypg_bis_ful_campaignservice_bw-ypg-bis-ful-campaignservice-bw Info [BW-Core] BWENGINE-300001 Process Engine version 5.14.0, build V25_hotfix004, 2019-2-12
2019 Nov 12 14:58:01:211 GMT -0500 BW.ypg_bis_ful_campaignservice_bw-ypg-bis-ful-campaignservice-bw Info [BW-User] UPSTREAM_REQUEST Job-1000 [ESBCore/Log/System/Log.process/Log Message]: <?xml version="1.0" encoding="UTF-8"?>
<ns0:LogEntry xmlns:ns0="urn:com.ypg.core.schemas.logging.v2">
<ns1:ContextHeader xmlns:ns1="urn:com.ypg.common.types.v1">
ns1:OriginatorIdWebServices</ns1:OriginatorId>
ns1:SystemApplicationNameypg.bis.ful.campaignservice.v1</ns1:SystemApplicationName>
ns1:SystemServiceNameCampaignService</ns1:SystemServiceName>
ns1:SystemOperationNameListCampaignProducts</ns1:SystemOperationName>
ns1:SystemTrackingId20d3dd47-fca7-4885-926a-e562cbb9fe42</ns1:SystemTrackingId>
ns1:SystemTimeCreated2019-11-12T14:58:01.197-05:00</ns1:SystemTimeCreated>
ns1:TibcoProcessId1000</ns1:TibcoProcessId>
</ns1:ContextHeader>
ns0:LogEntryRequest
ns0:LogLevelINFO</ns0:LogLevel>
ns0:MsgCodeUPSTREAM_REQUEST</ns0:MsgCode>
ns0:MessageRequest received</ns0:Message>
ns0:Payload<?xml version="1.0" encoding="UTF-8"?>
<Payload><urn:ListCampaignProductsRequest xmlns:urn="urn:com.ypg.bis.ful.campaignservice.v1" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn1="urn:com.ypg.common.service.types.v1" xmlns:urn2="urn:com.ypg.common.types.v1" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" timezone="+00:00" validateOnly="false"><urn1:ContextHeader><urn2:OriginatorId>WebServices</urn2:OriginatorId></urn1:ContextHeader><urn:MerchantId>623778</urn:MerchantId></urn:ListCampaignProductsRequest></Payload></ns0:Payload>
ns0:Timestamp2019-11-12T14:58:01.206-05:00</ns0:Timestamp>
</ns0:LogEntryRequest>
</ns0:LogEntry>

I used the below filter in logstash config file, but the multiline doesn't work:
grok {

  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{DATA:TZ} %{DATA:GMT} %{GREEDYDATA:ApplicationName} %{LOGLEVEL:LogLevel} \[%{DATA:Logger}\] (?<message>(.|\r|\n)*)"}

}

Any idea please ?

Thanks.

4 
25-Jul-2019 13:20:42.634 WARNING [xxxxxxxx.xxx.com-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [ELMS DB connection pool connection closer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
5

above is an example of one of the multiline logs that I'm trying to show in one log ..

6

Please start your own thread for this.

7

Yes, I've created a thread but no answer till now:

Thanks.

8

Archie, please present your configuration too. Remember that you can try your own to adjust it to your log lines using the https://grokdebug.herokuapp.com/

9 
filebeat.inputs:
- type: log
  enabled: true
  paths:
      - /var/log/*.log
      - /var/log/*messages*
      - /var/log/secure
      - /var/apache-tomcat-8.5.23/logs/catalina*
      - /var/apache-tomcat-8.5.23/logs/*.log
### Multiline options
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#  multiline.pattern: ^[0-9]{2}-(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Sept|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)-[0-9]{4}
# multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
#multiline.pattern: ^[0-9]{2}-(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Sept|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)-[0-9]{4}
#multiline.negate: true
#multiline.match: after
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "xxxxx"
output.logstash:
  hosts: ["xxxxxx"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
xpack.monitoring.elasticsearch.hosts: ["https://xxxxxx"]
10

I have not tried to create one on my own . I have been trying to use the ones from the Elasticsearch documentation

11

This is the last one I have just tried but I have tried all of the ones on the Elasticsearch multiline page

12

Here is relevant documentation: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html

Your regex need to match to the beginning of the first line of every multiline log record, e.g. (^date_in_right_format).

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.