Filebeat not indetifying multiple line logs

Elastic Stack Beats
1

I have some springboot logs that start like this:

2018-09-11 09:18:42.968 INFO 12036 ---

And I created a pattern in the filebeat.yml file to consider multiple line logs as part of the previous line like this:

#=========================== Filebeat inputs =============================

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

  • type: log

    Change to true to enable this input configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs*

    • /home/myuser/project/app/logs/project.trace.log

    Exclude lines. A list of regular expressions to match. It drops the lines that are

    matching any regular expression from the list.

    #exclude_lines: ['^DBG']

    Include lines. A list of regular expressions to match. It exports the lines that are

    matching any regular expression from the list.

    #include_lines: ['^ERR', '^WARN']

    Exclude files. A list of regular expressions to match. Filebeat drops the files that

    are matching any regular expression from the list. By default, no files are dropped.

    #exclude_files: ['.gz$']

    Optional additional fields. These fields can be freely picked

    to add additional information to the crawled log files for filtering

    #fields:

    level: debug

    review: 1

    Multiline options

    Multiline can be used for log messages spanning multiple lines. This is common

    for Java Stack Traces or C-Line Continuation

    The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

    multiline.pattern: ^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*

    Defines if the pattern set under pattern should be negated or not. Default is false.

    multiline.negate: true

    Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

    that was (not) matched before or after or as long as a pattern is not matched based on negate.

    Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

    multiline.match: after

    On kibana I can see just one log being generated, as if filebeat is concatenating all my logs file.
    I'm sending this logs to logstash and if I remove these multiline.pattern option the logs are being generated correctly, besides multiple line logs.
    I don't see anything wrong with my patterns, could anyone please help me on this?

2

Hi @Nelson_Ferreira_Juni and welcome :slight_smile:

I don't think the pattern you are using is a valid regular expression for filebeat. Filebeat uses golang regular expressions, that follow RE2 syntax. You could start trying with a simple regular expression to match the year, then continue polishing it.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.