# Multi line logs Javastack trace not showing up correctly

**URL:** https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903
**Category:** Beats
**Tags:** filebeat
**Created:** [July 28, 2020, 12:28pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903 "2020-07-28T12:28:03Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![Archie\_Crawford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/archie_crawford/32/62732_2.png) [@Archie\_Crawford](https://discuss.elastic.co/u/Archie_Crawford)
#### Post date: [July 28, 2020, 12:28pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/1 "2020-07-28T12:28:03Z")

</div>

The java stack trace logs are not coming in as multi-lines they are still coming in as separate lines

Filebeat Config

```auto
      - /var/***/catalina*
     - /var/***/*.log
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:![image|689x269](upload://1gTJ1sNVcJL77EQrk6NvggwAlyE.png) `

```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/8/08eb32555e110ed7612f3baf5ed1a80dea039018.png)

---

<div class="post-metadata">

### Author: ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [July 28, 2020, 3:07pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/2 "2020-07-28T15:07:46Z")

</div>

Could you please correct the formatting using code tags? Also, please provide sample logs here.

---

<div class="post-metadata">

### Author: ![Arezki76](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@Arezki76](https://discuss.elastic.co/u/Arezki76)
#### Post date: [July 28, 2020, 3:28pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/3 "2020-07-28T15:28:11Z")

</div>

> [@mtojek](#):
>
> code tags

Hello mtojek,

I have the same issue, here under my logs:  
2019 May 08 08:27:37:283 GMT -0400 BW.ypg\_bis\_ful\_campaignservice\_bw-ypg-bis-ful-campaignservice-bw Info [BW-Core] BWENGINE-300001 Process Engine version 5.14.0, build V25\_hotfix004, 2019-2-12  
2019 Nov 12 14:58:01:211 GMT -0500 BW.ypg\_bis\_ful\_campaignservice\_bw-ypg-bis-ful-campaignservice-bw Info [BW-User] UPSTREAM\_REQUEST Job-1000 [ESBCore/Log/System/Log.process/Log Message]: \<?xml version="1.0" encoding="UTF-8"?\>  
\<ns0:LogEntry xmlns:ns0="urn:com.ypg.core.schemas.logging.v2"\>  
\<ns1:ContextHeader xmlns:ns1="urn:com.ypg.common.types.v1"\>  
ns1:OriginatorIdWebServices\</ns1:OriginatorId\>  
ns1:SystemApplicationNameypg.bis.ful.campaignservice.v1\</ns1:SystemApplicationName\>  
ns1:SystemServiceNameCampaignService\</ns1:SystemServiceName\>  
ns1:SystemOperationNameListCampaignProducts\</ns1:SystemOperationName\>  
ns1:SystemTrackingId20d3dd47-fca7-4885-926a-e562cbb9fe42\</ns1:SystemTrackingId\>  
ns1:SystemTimeCreated2019-11-12T14:58:01.197-05:00\</ns1:SystemTimeCreated\>  
ns1:TibcoProcessId1000\</ns1:TibcoProcessId\>  
\</ns1:ContextHeader\>  
ns0:LogEntryRequest  
ns0:LogLevelINFO\</ns0:LogLevel\>  
ns0:MsgCodeUPSTREAM\_REQUEST\</ns0:MsgCode\>  
ns0:MessageRequest received\</ns0:Message\>  
ns0:Payload\<?xml version="1.0" encoding="UTF-8"?\>  
\<Payload\>\<urn:ListCampaignProductsRequest xmlns:urn="urn:com.ypg.bis.ful.campaignservice.v1" xmlns:soapenv="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)" xmlns:urn1="urn:com.ypg.common.service.types.v1" xmlns:urn2="urn:com.ypg.common.types.v1" xmlns:SOAP-ENV="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)" timezone="+00:00" validateOnly="false"\>\<urn1:ContextHeader\>\<urn2:OriginatorId\>WebServices\</urn2:OriginatorId\>\</urn1:ContextHeader\>\<urn:MerchantId\>623778\</urn:MerchantId\>\</urn:ListCampaignProductsRequest\>\</Payload\>\</ns0:Payload\>  
ns0:Timestamp2019-11-12T14:58:01.206-05:00\</ns0:Timestamp\>  
\</ns0:LogEntryRequest\>  
\</ns0:LogEntry\>

I used the below filter in logstash config file, but the multiline doesn't work:  
grok {

```
  match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{DATA:TZ} %{DATA:GMT} %{GREEDYDATA:ApplicationName} %{LOGLEVEL:LogLevel} \[%{DATA:Logger}\] (?<message>(.|\r|\n)*)"}

```

}

Any idea please ?

Thanks.

---

<div class="post-metadata">

### Author: ![Archie\_Crawford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/archie_crawford/32/62732_2.png) [@Archie\_Crawford](https://discuss.elastic.co/u/Archie_Crawford)
#### Post date: [July 28, 2020, 5:14pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/4 "2020-07-28T17:14:34Z")

</div>

```auto
25-Jul-2019 13:20:42.634 WARNING [xxxxxxxx.xxx.com-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [ELMS DB connection pool connection closer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)

```

---

<div class="post-metadata">

### Author: ![Archie\_Crawford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/archie_crawford/32/62732_2.png) [@Archie\_Crawford](https://discuss.elastic.co/u/Archie_Crawford)
#### Post date: [July 28, 2020, 5:15pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/5 "2020-07-28T17:15:40Z")

</div>

above is an example of one of the multiline logs that I'm trying to show in one log ..

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 28, 2020, 10:23pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/6 "2020-07-28T22:23:18Z")

</div>

Please start your own thread for this.

---

<div class="post-metadata">

### Author: ![Arezki76](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@Arezki76](https://discuss.elastic.co/u/Arezki76)
#### Post date: [July 28, 2020, 11:50pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/7 "2020-07-28T23:50:40Z")

</div>

Yes, I've created a thread but no answer till now:

> [@Logstash - filebeat](https://discuss.elastic.co/t/logstash-filebeat/242216):
>
> Hi, I'm new in ELK and I'm trying to parse log file as follow: 2019 Nov 12 14:58:01:211 GMT -0500 BW.ypg\_bis\_ful\_campaignservice\_bw-ypg-bis-ful-campaignservice-bw Info [BW-User] UPSTREAM\_REQUEST Job-1000 [ESBCore/Log/System/Log.process/Log Message]: \<?xml version="1.0" encoding="UTF-8"?\> \<ns0:LogEntry xmlns:ns0="urn:com.ypg.core.schemas.logging.v2"\> \<ns1:ContextHeader xmlns:ns1="urn:com.ypg.common.types.v1"\> ns1:OriginatorIdWebServices\</ns1:OriginatorId\> ns1:SystemApplicationNameypg.bis.fu…

Thanks.

---

<div class="post-metadata">

### Author: ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [July 29, 2020, 7:25am UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/8 "2020-07-29T07:25:52Z")

</div>

Archie, please present your configuration too. Remember that you can try your own to adjust it to your log lines using the [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/)

---

<div class="post-metadata">

### Author: ![Archie\_Crawford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/archie_crawford/32/62732_2.png) [@Archie\_Crawford](https://discuss.elastic.co/u/Archie_Crawford)
#### Post date: [July 29, 2020, 12:00pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/9 "2020-07-29T12:00:11Z")

</div>

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
      - /var/log/*.log
      - /var/log/*messages*
      - /var/log/secure
      - /var/apache-tomcat-8.5.23/logs/catalina*
      - /var/apache-tomcat-8.5.23/logs/*.log
### Multiline options
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
# multiline.pattern: ^[0-9]{2}-(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Sept|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)-[0-9]{4}
# multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
#multiline.pattern: ^[0-9]{2}-(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Sept|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)-[0-9]{4}
#multiline.negate: true
#multiline.match: after
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "xxxxx"
output.logstash:
  hosts: ["xxxxxx"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
xpack.monitoring.elasticsearch.hosts: ["https://xxxxxx"]

```

---

<div class="post-metadata">

### Author: ![Archie\_Crawford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/archie_crawford/32/62732_2.png) [@Archie\_Crawford](https://discuss.elastic.co/u/Archie_Crawford)
#### Post date: [July 29, 2020, 12:01pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/10 "2020-07-29T12:01:59Z")

</div>

I have not tried to create one on my own . I have been trying to use the ones from the Elasticsearch documentation

---

<div class="post-metadata">

### Author: ![Archie\_Crawford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/archie_crawford/32/62732_2.png) [@Archie\_Crawford](https://discuss.elastic.co/u/Archie_Crawford)
#### Post date: [July 29, 2020, 12:03pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/11 "2020-07-29T12:03:02Z")

</div>

This is the last one I have just tried but I have tried all of the ones on the Elasticsearch multiline page

---

<div class="post-metadata">

### Author: ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [July 29, 2020, 1:51pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/12 "2020-07-29T13:51:16Z")

</div>

Here is relevant documentation: [https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)

Your regex need to match to the beginning of the first line of every multiline log record, e.g. (`^date_in_right_format`).

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 26, 2020, 3:51pm UTC](https://discuss.elastic.co/t/multi-line-logs-javastack-trace-not-showing-up-correctly/242903/13 "2020-08-26T15:51:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
