Struggling with Multiline

Good morning,

I'm (still) facing some problems with the filebeat multiline feature.

My filebeat.yml is the following:

prospectors:
     -
       paths:
          - /tmp/multiline*.log
       input_type: log
          multiline.pattern: '^\[|^[[:space:]]+|^Caused by:'
          multiline.negate: true
          multiline.match: before

My multiline.log

[Thread-360]:28.11.2017 09:26:26.254] ERROR [id:] commons.service.queue.MessageQueueSubscriber: listen() exception on receive, resetting
org.springframework.jms.IllegalStateException: Error Connecting MQXYIS; nested exception is com.ibm.msg.client.jms.DetailedIllegalStateException: 
         at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
         at java.lang.Thread.run(Thread.java:745)
Caused by: com.ibm.msg.client.jms.DetailedIllegalStateException:  Error Connecting MQXYIS
        at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:474)
        ... 4 common frames omitted
Caused by: com.ibm.mq.MQException: Error Connecting MQXYIS.
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:204)
       ... 16 common frames omitted

I've enabled the debug mode in filebeat in order to get the published event (last 2 lines).

[...]
  "input_type": "log",
  "message": "        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:204)",
  "offset": 15846,
  "source": "/tmp/multiline.log",
  "type": "log"
 }

 [...]
  "input_type": "log",
  "message": "        ... 16 common frames omitted",
  "offset": 15883,
  "source": "/tmp/multiline.log",
  "type": "log"
 }

I was expecting to get only 1 published event (all events separated with a \n).
Do I understand correctly or not?

My logback.xml pattern is:

 [%thread]:%d{dd.MM.yyyy HH:mm:ss.SSS}] %-5level [id:%mdc{-Request-ID}] %logger: %msg%n

Many thanks for your help

Please format your config and sample logs. Surround each block triple backticks.

Looks like you try to adapt the multiline config from the sample docs. Do log messages always start with a stream ID + timestamp? In this case I would match on the presense of thread ID via ^\[Thread-\d+\] (You will have to adapt the negate and match settings).

Hi @andrewkroh, @steffens,

Many thanks for your help.
Yes, as describe in the logback, the log message always start with a [Thread-ID] %DATE

Well, I'm trying to adapt my previous logstash config (which is working with multilines) to use the ingest node feature. (I'm using elk 5.6.4)

I've tried to set the pattern to ^\[Thread-\d+\] but I get the same result. Only the first line is inserted into elastic.

Maybe as useful information (encoding is utf-8 & carriage return is $):

file -bi /tmp/multiline.log 
text/plain; charset=utf-8

and

vi /tmp/multiline.log --> :set list
        [...]
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:204)$
        ... 16 common frames omitted$

can you post your complete filebeat configuration (feel free to remove comments)? Filebeat uses YAML, which is sensitive to indentation (always use 2 spaces, no tabs). Your config from first post looks completely off.

Using the pattern with sample logs + test script works quite well: https://play.golang.org/p/f5q4l87fN-

Hi @steffens,

Sure. Hereafter the config I'm using:

############################# Filebeat ######################################
filebeat:
#=========================== Filebeat prospectors =============================
  # List of prospectors to fetch data.
  prospectors:
    -
      paths:
        - /tmp/multiline*.log
      input_type: log
      fields:
         app_type: test
         app_client: test
         log_type: test
      multiline:
         pattern: '^\[Thread-\d+\]'
         negate: false
         match: after

#================================ Outputs =====================================
output:
#-------------------------- Elasticsearch output ------------------------------
  elasticsearch:
    hosts: ["elk:9200"]
    pipeline: "grok_test_log"
    index: "%{[fields.app_client]}-%{[fields.app_type]}-%{[fields.log_type]}-%{+yyyy.MM.dd}"

    template:
      path: "/etc/filebeat/filebeat.template.json"

#----------------------------- Debugging output --------------------------------
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: debug.log
  keepfiles: 3

I've tried with plenty different multiline patterns, but I always get the same result...

Thanks for your help
Greg

How do you start filebeat?

For testing can you do this:

$ rm -f /tmp/testreg.json; filebeat -e -v -c <pathto>/filebeat.yml -E filebeat.registry_file=/tmp/testreg.json -E output.elasticsearch.enabled=false -E output.console.pretty=true

This will pretty-print all events to console (instead of sending to elasticsearch), with a custom registry file (such that global state is not overwritten). You get multiline events written to console? I always get confused about match and negate settings. While testing You can try to change those from CLI via -E filebeat.prospectors.0.multiline='{negate: false, match: after}'.

Hi @steffens,

I was "tailing" the debug.log file all the time, thanks for sharing this nice way of testing!

Well, now I'm also really confused about the match & negate settings (I thought I've tried all the possible combinaison, but apparently not)!
I've changed to negate: true & match: after and it seems to work like a charm!

Many thanks for the time you spent (all of you)!
Greg

I think I've found the mistake I had...

The multiline indentation was probably incorrect! Thus, filebeat was interpreting the multiline as a field and was not performing the multiline... Sorry

  "fields": {
    "app_client": "test",
    "app_type": "test",
    "log_type": "test",
    "multiline": {
      "match": "after",
      "negate": true,
      "pattern": "^\\[Thread-\\d+\\]"
    }

Hi, I am facing issues while handling multiline codec in file beats,

I need to handle below-

app-node1 ERROR exception in executing method
at shared.util.rest.RESTConnector.executeHttpMethod(RESTConnector.java:193) [app-shared-ejb.jar:]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) [rt.jar:1.8.0_65]

When I am using below pattern, I am getting single lines in debug mode-

Filebeat config-

output.logstash:
hosts: ["0.0.0.0:5044"]
multiline.pattern: '^[app]'
multiline.negate: true
multiline.match: after

Output I am getting as

message:app-node1 ERROR exception in executing method
message: \t at shared.util.rest.RESTConnector.executeHttpMethod(RESTConnector.java:193) [app-shared-

Hi,

I was able to identify the issue. Its working fine for me.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.