Multiline not working properly when used along with exclude_lines

Elastic Stack Beats
1

We were trying to combine all the logs into a single event after excluding lines with a specific pattern. But it seems filebeat is appending the excluded lines in multiline and misses out other data leading to misbehavior. Not sure whether the pattern I've given is wrong. There is an issue related to this in github but it is closed without proper explanation. As per this answer by @ruflin, is it because exclude_lines is a processor and multiline is a reader?

We use filebeat-7.5.2 which is running on kubernetes 1.18.8

Here is the kubectl configmap for filebeat.yaml:

apiVersion: v1
data:
  filebeat.yaml: |-
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /logs/*.log
      exclude_lines: ['^Microsoft\.Data\.Tools\.Diagnostics\.Tracer (Warning|Verbose).*']
      multiline:
        pattern: '.*'
        negate: false
        match: after

    output.console:
      pretty: true

    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

    setup.template.settings:
      index.number_of_shards: 1

    setup.ilm.enabled: auto
    setup.ilm.rollover_alias: "filebeat"
    setup.ilm.pattern: "{now/d}-000001"

    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~
      - add_docker_metadata: ~
      - add_kubernetes_metadata: ~
      - add_fields:
          target: event
          fields:
            dataset: "sql-job"
kind: ConfigMap
metadata:
  creationTimestamp: "2020-08-28T11:51:28Z"
  name: sql-job-filebeat
  namespace: test

Example input:

Microsoft.Data.Tools.Diagnostics.Tracer Warning: 0 : 2020-09-03T15:23:37 : SqlStudioException.TraceException(Exception)
 Exception: System.ArgumentException: The property Microsoft.Data.Tools.Schema.Sql.CommandLine.CommandLineResources specified is not present in the resource AdditionalDeploymentContributorPaths_FriendlyName.
Parameter name: resourcePropertyName
Publishing to database 'devtestSystem' on server '1.2.3.4'.
Microsoft.Data.Tools.Diagnostics.Tracer Information: 0 : 2020-09-03T15:23:37 : Initialized ModelSchema - elements 149
Microsoft.Data.Tools.Diagnostics.Tracer Information: 19 : 2020-09-03T15:23:37 : IsolatedStorageFile probe success
Microsoft.Data.Tools.Diagnostics.Tracer Warning: 0 : 2020-09-03T15:23:37 : Logging Ambient Settings...
Microsoft.Data.Tools.Diagnostics.Tracer Warning: 0 : 2020-09-03T15:23:37 : DatabaseLockTimeout: 60000
Microsoft.Data.Tools.Diagnostics.Tracer Verbose: 0 : 2020-09-03T15:23:37 : Loading collation id 1078 af-ZA
Microsoft.Data.Tools.Diagnostics.Tracer Verbose: 1 : 2020-09-03T15:24:13 : ColumnResolver: #47626729 ResolvedDescriptor for column:
ResolvedDescriptor: [tableInsID]
Potentials(3):
        [SqlColumn : P1 - [dbo.tablexyzIns.tableInsID]]
        [SqlColumn : P1 - [dbo.tablexyzIns.tables]::PtableInsID]
        [SqlColumn : P1 - [dbo.tableIns.tables]::PtableInsID]
Microsoft.Data.Tools.Diagnostics.Tracer Verbose: 1 : 2020-09-03T15:24:13 : ColumnResolver: #32559816 Merging source without existing source, ExposedName = tables
Microsoft.Data.Tools.Diagnostics.Tracer Verbose: 1 : 2020-09-03T15:24:13 : ColumnResolver: #32559816 Added source:^M
        ExposedName = tables^M
        Target=(ISqlColumnSource)dbo.tablexyzIns^M
        Affinity = 1^M
        AmbiguousSchema = False^M
        IsCTE = False^M
        IsMirror = False
Microsoft.Data.Tools.Diagnostics.Tracer Verbose: 1 : 2020-09-03T15:24:13 : ColumnResolver: #20099428 ResolvedDescriptor for column:
ResolvedDescriptor: [tableExecuteModeID]
Potentials(7):
        [SqlColumn : P1 - [dbo.tablexyzIns.tableIns]::PtableExecuteModeID]
        [SqlColumn : P1 - [dbo.tableIns.tableExecuteModeID]]
        [SqlColumn : P1 - [dbo.tableIns.tableIns]::PtableExecuteModeID]
        [SqlColumn : P1 - [dbo.tableDef.tableIns]::PtableExecuteModeID]
        [SqlColumn : P1 - [TinyEnum.LockedStatus.tableIns]::PtableExecuteModeID]
        [SqlColumn : P1 - [TinyEnum.VisibilityStatus.tableIns]::PtableExecuteModeID]
        [SqlColumn : P1 - [TinyEnum.tableExecuteMode.tableIns]::PtableExecuteModeID]
Microsoft.Data.Tools.Diagnostics.Tracer Verbose: 0 : 2020-09-03T15:24:50 : Executing Step 16, Not Tracked, Type 'CreateElementStep', Section 'User', Operation '1', Ignorable Errors 'None' (SqlCheckConstraint), Script is as follows:

IF @@ERROR <> 0
   AND @@TRANCOUNT > 0
    BEGIN
        ROLLBACK;
    END

IF @@TRANCOUNT = 0
    BEGIN
        INSERT  INTO #tmpErrors (Error)
        VALUES                 (1);
        BEGIN TRANSACTION;
    END

Microsoft.Data.Tools.Diagnostics.Tracer Verbose: 0 : 2020-09-03T15:24:50 : Executing Step 17, Not Tracked, Type 'SqlEndTransactionStep', Section 'User', Script is as follows:

IF EXISTS (SELECT *
           FROM   #tmpErrors)
    ROLLBACK;
GO
IF @@TRANCOUNT > 0
    BEGIN
        PRINT N'The transacted portion of the database update succeeded.';
        COMMIT TRANSACTION;
    END
ELSE
    PRINT N'The transacted portion of the database update failed.';
Available Column Sources:
ColumnSource tables (affinity = 1) with no known columns
ColumnSource tableIns (affinity = 1) with no known columns
ColumnSource tableDef (affinity = 1) with no known columns
ColumnSource LockedStatus (affinity = 1) with no known columns
ColumnSource VisibilityStatus (affinity = 1) with no known columns
ColumnSource tableExecuteMode (affinity = 1) with no known columns
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.