Struggling with Multiline

rosselg · November 28, 2017, 10:05am

Good morning,

I'm (still) facing some problems with the filebeat multiline feature.

My filebeat.yml is the following:

prospectors:
     -
       paths:
          - /tmp/multiline*.log
       input_type: log
          multiline.pattern: '^\[|^[[:space:]]+|^Caused by:'
          multiline.negate: true
          multiline.match: before

My multiline.log

[Thread-360]:28.11.2017 09:26:26.254] ERROR [id:] commons.service.queue.MessageQueueSubscriber: listen() exception on receive, resetting
org.springframework.jms.IllegalStateException: Error Connecting MQXYIS; nested exception is com.ibm.msg.client.jms.DetailedIllegalStateException: 
         at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
         at java.lang.Thread.run(Thread.java:745)
Caused by: com.ibm.msg.client.jms.DetailedIllegalStateException:  Error Connecting MQXYIS
        at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:474)
        ... 4 common frames omitted
Caused by: com.ibm.mq.MQException: Error Connecting MQXYIS.
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:204)
       ... 16 common frames omitted

I've enabled the debug mode in filebeat in order to get the published event (last 2 lines).

[...]
  "input_type": "log",
  "message": "        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:204)",
  "offset": 15846,
  "source": "/tmp/multiline.log",
  "type": "log"
 }

 [...]
  "input_type": "log",
  "message": "        ... 16 common frames omitted",
  "offset": 15883,
  "source": "/tmp/multiline.log",
  "type": "log"
 }

I was expecting to get only 1 published event (all events separated with a \n).
Do I understand correctly or not?

My logback.xml pattern is:

 [%thread]:%d{dd.MM.yyyy HH:mm:ss.SSS}] %-5level [id:%mdc{-Request-ID}] %logger: %msg%n

Many thanks for your help

andrewkroh · November 28, 2017, 10:34am

Please format your config and sample logs. Surround each block triple backticks.

steffens · November 28, 2017, 2:46pm

Looks like you try to adapt the multiline config from the sample docs. Do log messages always start with a stream ID + timestamp? In this case I would match on the presense of thread ID via ^\[Thread-\d+\] (You will have to adapt the negate and match settings).

rosselg · November 28, 2017, 3:16pm

Hi @andrewkroh, @steffens,

Many thanks for your help.
Yes, as describe in the logback, the log message always start with a [Thread-ID] %DATE

Well, I'm trying to adapt my previous logstash config (which is working with multilines) to use the ingest node feature. (I'm using elk 5.6.4)

I've tried to set the pattern to ^\[Thread-\d+\] but I get the same result. Only the first line is inserted into elastic.

Maybe as useful information (encoding is utf-8 & carriage return is $):

file -bi /tmp/multiline.log 
text/plain; charset=utf-8

and

vi /tmp/multiline.log --> :set list
        [...]
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:204)$
        ... 16 common frames omitted$

steffens · November 29, 2017, 1:26pm

can you post your complete filebeat configuration (feel free to remove comments)? Filebeat uses YAML, which is sensitive to indentation (always use 2 spaces, no tabs). Your config from first post looks completely off.

Using the pattern with sample logs + test script works quite well: https://play.golang.org/p/f5q4l87fN-

rosselg · November 29, 2017, 2:03pm

Hi @steffens,

Sure. Hereafter the config I'm using:

############################# Filebeat ######################################
filebeat:
#=========================== Filebeat prospectors =============================
  # List of prospectors to fetch data.
  prospectors:
    -
      paths:
        - /tmp/multiline*.log
      input_type: log
      fields:
         app_type: test
         app_client: test
         log_type: test
      multiline:
         pattern: '^\[Thread-\d+\]'
         negate: false
         match: after

#================================ Outputs =====================================
output:
#-------------------------- Elasticsearch output ------------------------------
  elasticsearch:
    hosts: ["elk:9200"]
    pipeline: "grok_test_log"
    index: "%{[fields.app_client]}-%{[fields.app_type]}-%{[fields.log_type]}-%{+yyyy.MM.dd}"

    template:
      path: "/etc/filebeat/filebeat.template.json"

#----------------------------- Debugging output --------------------------------
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: debug.log
  keepfiles: 3

I've tried with plenty different multiline patterns, but I always get the same result...

Thanks for your help
Greg

steffens · November 29, 2017, 2:20pm

How do you start filebeat?

For testing can you do this:

$ rm -f /tmp/testreg.json; filebeat -e -v -c <pathto>/filebeat.yml -E filebeat.registry_file=/tmp/testreg.json -E output.elasticsearch.enabled=false -E output.console.pretty=true

This will pretty-print all events to console (instead of sending to elasticsearch), with a custom registry file (such that global state is not overwritten). You get multiline events written to console? I always get confused about match and negate settings. While testing You can try to change those from CLI via -E filebeat.prospectors.0.multiline='{negate: false, match: after}'.

rosselg · November 30, 2017, 11:29am

Hi @steffens,

I was "tailing" the debug.log file all the time, thanks for sharing this nice way of testing!

Well, now I'm also really confused about the match & negate settings (I thought I've tried all the possible combinaison, but apparently not)!
I've changed to negate: true & match: after and it seems to work like a charm!

Many thanks for the time you spent (all of you)!
Greg

rosselg · November 30, 2017, 11:47am

I think I've found the mistake I had...

The multiline indentation was probably incorrect! Thus, filebeat was interpreting the multiline as a field and was not performing the multiline... Sorry

  "fields": {
    "app_client": "test",
    "app_type": "test",
    "log_type": "test",
    "multiline": {
      "match": "after",
      "negate": true,
      "pattern": "^\\[Thread-\\d+\\]"
    }

Sumit_Sijaria · December 1, 2017, 5:43am

Hi, I am facing issues while handling multiline codec in file beats,

I need to handle below-

app-node1 ERROR exception in executing method
at shared.util.rest.RESTConnector.executeHttpMethod(RESTConnector.java:193) [app-shared-ejb.jar:]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) [rt.jar:1.8.0_65]

When I am using below pattern, I am getting single lines in debug mode-

Filebeat config-

output.logstash:
hosts: ["0.0.0.0:5044"]
multiline.pattern: '^[app]'
multiline.negate: true
multiline.match: after

Output I am getting as

message:app-node1 ERROR exception in executing method
message: \t at shared.util.rest.RESTConnector.executeHttpMethod(RESTConnector.java:193) [app-shared-

Sumit_Sijaria · December 1, 2017, 7:00am

Hi,

I was able to identify the issue. Its working fine for me.

system · December 29, 2017, 7:00am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
FIlebeat multiline pattern Beats filebeat	2	529	January 23, 2018
Filebeat not honoring multiline configuration Beats filebeat	3	655	May 25, 2017
Issues with multiline Beats filebeat	6	1461	August 17, 2017
Error with multiline codec Beats filebeat	3	1402	August 3, 2017
Filebeat 6.0 Multiline issue Beats filebeat	17	1509	December 27, 2017

Struggling with Multiline

Related topics