Filebeat Multiline Patterns not working for us

Elastic Stack Beats
1

I wish to parse the tomcat logs. But I am facing problem in multiline handling in Filebeat. The logs are not parsed as per the requirement. Getting grokparse failures. Upon checking, I could see that the line starts doesn't with the date are not appended to the lines starts with the date.

We are using the following multiline settings

   - type: log
   multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{2}-[A-z]{3}-[0-9]{4}|^[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
   multiline.negate: true
   multiline.match: after

The log file of the following format :

23-Oct-2018 13:19:41.677 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.5.11
23-Oct-2018 13:19:41.757 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Jan 10 2017 21:02:52 UTC
2018-10-23 15:00:21.464 http-nio-8085-exec-22 o.h.engine.loading.internal.CollectionLoadContext

15:00:21.464 http-nio-8085-exec-22 o.h.engine.loading.internal.CollectionLoadContext

According the multiline settings the message either starts with "Year-month-monthday", "Monthday-month-year" or "Hour:minute:seconds".

But the messages starts with the folliowing :

            "message" => "\t\t\t\tbegin",
            "message" => "\t\t\t\tselect nextval ('audit_trail_id_seq')",
            "message" => "Hibernate: select nextval ('audit_trail_id_seq')",
            "message" => "\t\t\t\tSequence value obtained: 97451",
            "message" => "\t\t\t\tGetting ready to invoke write method on the destination object.  Dest Obj: ActionEntity, Dest value: Fetch ticket count Success USER ID=69"
            "message" => "\t\t\t\tResult set row: 49",
            "message" => "\t\t\t\tResult row: ",

The line starts doesn't with the date are not appended to the lines starts with the date and time.

I have tried with the following multiline settings. Accoring to the setings the message starts with space,message starts with space,messgage starts with single white space character. This includes space, tab etc, messgage starts with at and messgage starts with should be appended to the line which doesn’t match the pattern.

multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:|^[[:space:]]|^\s'
multiline.negate: false
multiline.match: after

But still the issue is not resolved. Getting the following message :

             "message" => "\t\t\t\tbegin",
             "message" => "\t\t\t\tselect nextval ('audit_trail_id_seq')",
             "message" => "Caused by: com.sun.mail.util.MailConnectException: Couldn't connect to host, port: x.x.x.x, 25; timeout -1",

Please let us know what is the error with multiline pattern which we are using. Awaitng the reply.

2

Can you share your complete inputs section in your filebeat configuration file?

3

The filebeat configuration file "filebeat.yml" section starts with "Filebeat prospectors".

filebeat.prospectors:
- type: log 
  enabled: true
     - /var/log/*.log


  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{2}-[A-z]{3}-[0-9]{4}|^[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
  multiline.negate: true
  multiline.match: after

  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
  multiline.negate: false
  multiline.match: after

This is the "Filebeat prospectors" section. I have avoided the commented entries.

4

You can not have multiple multiline configurations per prospector. If you have some files that require special handling, create separate prospector configs for them, like:

filebeat.prospectors:
- type: log
  paths:
    - file1.log

  multiline....

- type: log
  paths:
    - file2.log

  multiline....
5

I have commented the following multiline configuration.

multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after

Now, there is only following multiline configuration.

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{2}-[A-z]{3}-[0-9]{4}|^[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
multiline.negate: true
multiline.match: after

But the issue still persists.

6

Can you please post your complete configuration file? Including comments.

Give your sample logs this regex should work: ^((\d{2}-[A-Z][a-z]{2}-\d{4})|(\d{4}-\d{2}-\d{2})|(\d{2}:\d{2}:\d{2}.\d{3}))

See this program for testing (just press 'run'): https://play.golang.org/p/8y2-O3cdMsj

7

Please check the complete filebeat configuration :

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- type: log

  # Change to true to enable this prospector configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{2}-[A-z]{3}-[0-9]{4}|^[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
  multiline.negate: true
  multiline.match: after


  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: /etc/filebeat/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
setup.dashboards.enabled: true

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================= Elastic Cloud ==================================

# These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
 # hosts: ["10.10.x.x:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "elastic121"

#----------------------------- Logstash output --------------------------------

output.logstash:
  # The Logstash hosts
  hosts: ["10.10.x.x:5044"]
#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

I have tried with following multiline configurations.

 multiline.pattern: '^\d{4}-\d{2}-\d{2}|^\d{2}:\d{2}:\d{2}.\d{3}|^\d{2}-[A-z]{3}-\d{4}'
 multiline.pattern: '^((\d{2}-[A-Z][a-z]{3}-\d{4})|(\d{4}-\d{2}-\d{2})|(\d{2}:\d{2}:\d{2}.\d{3}))'

But no luck. Kindly assist.

8

I have excluded the sections Xpack Monitoring, kibana and some of the comments from the complete filebeat configuration due character limit on the discuss form.

9

I have checked the following patterns online via "Go Playground".

var pattern = `^\d{4}-\d{2}-\d{2}|^\d{2}:\d{2}:\d{2}.\d{3}|^\d{2}-[A-z]{3}-\d{4}`
var pattern = `^((\d{2}-[A-Z][a-z]{2}-\d{4})|(\d{4}-\d{2}-\d{2})|(\d{2}:\d{2}:\d{2}.\d{3}))`
var pattern = `^[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{2}-[A-z]{3}-[0-9]{4}|^[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}`

The messages match all the specified configuration. But, none of the patterns working while using with filebeat multiline configuration. The logs are not appending to the previous lines.

Kindly assist.

10 
1. 2018-10-30 14:00:04.182 ERROR 20334 --- [pool-7-thread-1] javax.xml.messaging.saaj.client.p2p      : SAAJ0009: Message send failed",

2. 0.23544058802248913|STARTED COMMUNIATING WITH PG |Tue Oct 30 14:00:00 IST 2018||TRY 1 FAILED :ERROR092|:TRYING WITH 2nd Interation|Tue Oct 30 14:00:04 IST 2018|TRY 2 FAILED :ERROR092|:TRYING WITH 3nd Interation|Tue Oct 30 14:00:04 IST 2018|TRY 3 FAILED :ERROR092|{getTransactionTokenReturn=null, ERROR=ERROR092}",

According to the multline configuration, the second line should be appended to the first line. But, it is not appending. This is the multiline configuration :

  multiline.pattern: '^((\d{2}-[A-Z][a-z]{3}-\d{4})|(\d{4}-\d{2}-\d{2})|(\d{2}:\d{2}:\d{2}.\d{3}))'
  multiline.negate: true
  multiline.match: after

Please assist.

11

You can try these patterns with the communities filebeat-multiline-tester + different settings for match and negate.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.