# Struggling with Multiline
**URL:** https://discuss.elastic.co/t/struggling-with-multiline/109357
**Category:** Beats
**Tags:** filebeat
**Created:** [November 28, 2017, 10:05am UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357 "2017-11-28T10:05:58Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@rosselg](https://discuss.elastic.co/u/rosselg)
#### Post date: [November 28, 2017, 10:05am UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/1 "2017-11-28T10:05:58Z")
Good morning,
I'm (still) facing some problems with the filebeat multiline feature.
My filebeat.yml is the following:
```auto
prospectors:
-
paths:
- /tmp/multiline*.log
input_type: log
multiline.pattern: '^\[|^[[:space:]]+|^Caused by:'
multiline.negate: true
multiline.match: before
```
My multiline.log
```auto
[Thread-360]:28.11.2017 09:26:26.254] ERROR [id:] commons.service.queue.MessageQueueSubscriber: listen() exception on receive, resetting
org.springframework.jms.IllegalStateException: Error Connecting MQXYIS; nested exception is com.ibm.msg.client.jms.DetailedIllegalStateException:
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.ibm.msg.client.jms.DetailedIllegalStateException: Error Connecting MQXYIS
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:474)
... 4 common frames omitted
Caused by: com.ibm.mq.MQException: Error Connecting MQXYIS.
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:204)
... 16 common frames omitted
```
I've enabled the debug mode in filebeat in order to get the published event (last 2 lines).
```auto
[...]
"input_type": "log",
"message": " at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:204)",
"offset": 15846,
"source": "/tmp/multiline.log",
"type": "log"
}
[...]
"input_type": "log",
"message": " ... 16 common frames omitted",
"offset": 15883,
"source": "/tmp/multiline.log",
"type": "log"
}
```
I was expecting to get only 1 published event (all events separated with a \n).
Do I understand correctly or not?
My logback.xml pattern is:
```auto
[%thread]:%d{dd.MM.yyyy HH:mm:ss.SSS}] %-5level [id:%mdc{-Request-ID}] %logger: %msg%n
```
Many thanks for your help
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [November 28, 2017, 10:34am UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/2 "2017-11-28T10:34:42Z")
Please format your config and sample logs. Surround each block [triple backticks](https://help.github.com/articles/basic-writing-and-formatting-syntax/#quoting-code).
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [November 28, 2017, 2:46pm UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/3 "2017-11-28T14:46:25Z")
Looks like you try to adapt the multiline config from the sample docs. Do log messages always start with a stream ID + timestamp? In this case I would match on the presense of thread ID via `^\[Thread-\d+\]` (You will have to adapt the negate and match settings).
---
### Author:  [@rosselg](https://discuss.elastic.co/u/rosselg)
#### Post date: [November 28, 2017, 3:16pm UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/4 "2017-11-28T15:16:20Z")
Hi @andrewkroh, @steffens,
Many thanks for your help.
Yes, as describe in the logback, the log message always start with a [Thread-ID] %DATE
Well, I'm trying to adapt my previous logstash config (which is working with multilines) to use the ingest node feature. (I'm using elk 5.6.4)
I've tried to set the pattern to `^\[Thread-\d+\]` but I get the same result. Only the first line is inserted into elastic.
Maybe as useful information (encoding is `utf-8` & carriage return is `$`):
```auto
file -bi /tmp/multiline.log
text/plain; charset=utf-8
```
and
```auto
vi /tmp/multiline.log --> :set list
[...]
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:204)$
... 16 common frames omitted$
```
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [November 29, 2017, 1:26pm UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/5 "2017-11-29T13:26:45Z")
can you post your complete filebeat configuration (feel free to remove comments)? Filebeat uses YAML, which is sensitive to indentation (always use 2 spaces, no tabs). Your config from first post looks completely off.
Using the pattern with sample logs + test script works quite well: [https://play.golang.org/p/f5q4l87fN-](https://play.golang.org/p/f5q4l87fN-)
---
### Author:  [@rosselg](https://discuss.elastic.co/u/rosselg)
#### Post date: [November 29, 2017, 2:03pm UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/6 "2017-11-29T14:03:16Z")
Hi @steffens,
Sure. Hereafter the config I'm using:
```auto
############################# Filebeat ######################################
filebeat:
#=========================== Filebeat prospectors =============================
# List of prospectors to fetch data.
prospectors:
-
paths:
- /tmp/multiline*.log
input_type: log
fields:
app_type: test
app_client: test
log_type: test
multiline:
pattern: '^\[Thread-\d+\]'
negate: false
match: after
#================================ Outputs =====================================
output:
#-------------------------- Elasticsearch output ------------------------------
elasticsearch:
hosts: ["elk:9200"]
pipeline: "grok_test_log"
index: "%{[fields.app_client]}-%{[fields.app_type]}-%{[fields.log_type]}-%{+yyyy.MM.dd}"
template:
path: "/etc/filebeat/filebeat.template.json"
#----------------------------- Debugging output --------------------------------
logging.level: debug
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: debug.log
keepfiles: 3
```
I've tried with plenty different multiline patterns, but I always get the same result...
Thanks for your help
Greg
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [November 29, 2017, 2:20pm UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/7 "2017-11-29T14:20:09Z")
How do you start filebeat?
For testing can you do this:
```auto
$ rm -f /tmp/testreg.json; filebeat -e -v -c /filebeat.yml -E filebeat.registry_file=/tmp/testreg.json -E output.elasticsearch.enabled=false -E output.console.pretty=true
```
This will pretty-print all events to console (instead of sending to elasticsearch), with a custom registry file (such that global state is not overwritten). You get multiline events written to console? I always get confused about `match` and `negate` settings. While testing You can try to change those from CLI via `-E filebeat.prospectors.0.multiline='{negate: false, match: after}'`.
---
### Author:  [@rosselg](https://discuss.elastic.co/u/rosselg)
#### Post date: [November 30, 2017, 11:29am UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/8 "2017-11-30T11:29:01Z")
Hi @steffens,
I was "tailing" the debug.log file all the time, thanks for sharing this nice way of testing!
Well, now I'm also really confused about the match & negate settings (I thought I've tried all the possible combinaison, but apparently not)!
I've changed to `negate: true` & `match: after` and it seems to work like a charm!
Many thanks for the time you spent (all of you)!
Greg
---
### Author:  [@rosselg](https://discuss.elastic.co/u/rosselg)
#### Post date: [November 30, 2017, 11:47am UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/9 "2017-11-30T11:47:21Z")
I think I've found the mistake I had...
The multiline indentation was probably incorrect! Thus, filebeat was interpreting the multiline as a field and was not performing the multiline... Sorry
```auto
"fields": {
"app_client": "test",
"app_type": "test",
"log_type": "test",
"multiline": {
"match": "after",
"negate": true,
"pattern": "^\\[Thread-\\d+\\]"
}
```
---
### Author:  [@Sumit\_Sijaria](https://discuss.elastic.co/u/Sumit_Sijaria)
#### Post date: [December 1, 2017, 5:43am UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/10 "2017-12-01T05:43:21Z")
Hi, I am facing issues while handling multiline codec in file beats,
I need to handle below-
app-node1 ERROR exception in executing method
at shared.util.rest.RESTConnector.executeHttpMethod(RESTConnector.java:193) [app-shared-ejb.jar:]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) [rt.jar:1.8.0\_65]
When I am using below pattern, I am getting single lines in debug mode-
Filebeat config-
output.logstash:
hosts: ["0.0.0.0:5044"]
multiline.pattern: '^[app]'
multiline.negate: true
multiline.match: after
Output I am getting as
message:app-node1 ERROR exception in executing method
message: \t at shared.util.rest.RESTConnector.executeHttpMethod(RESTConnector.java:193) [app-shared-
---
### Author:  [@Sumit\_Sijaria](https://discuss.elastic.co/u/Sumit_Sijaria)
#### Post date: [December 1, 2017, 7:00am UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/11 "2017-12-01T07:00:10Z")
Hi,
I was able to identify the issue. Its working fine for me.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 29, 2017, 7:00am UTC](https://discuss.elastic.co/t/struggling-with-multiline/109357/12 "2017-12-29T07:00:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.