Filebeat multiline config not working

[DPattee]

I have a 3rd party app that spits out a text file with multiple lines for a single event. An event has a consistent start line and an end line.

I have tried filebeat configurations that grab everything after a specific regex, everything before a specific regex, and everything between two regexes.

I've toggled 'negate' off and on and switched the main pattern from my start to my end patterns.

I've changed my regex from being just a match at the start of a line, to being that start of the line match plus a .* to grab the entire line.

None of these changes "do anything" - I still end up with every line of the file as a distinct record in elastic.

My yaml is valid, I double checked indentation with yamllint and filebeat is loading and running everything else (including the 'add field' processor immediately below the multiline part that it seems to be ignoring.

Even though I'm using some of the predefined modules for other files, I've just defined these two particular log patterns directly in my filebeat.yml. The relevant top portion (beyond this is just the kibana/elastic setup portions)

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 300s

# Extra inputs
filebeat.inputs:
- type: filestream
  paths:
    - /var/log/apcupsd*.events
  processors:
    - add_fields:
        target: apc
        fields:
          source: apcupsd-event
- type: filestream
  paths:
    - /var/log/apcupsd*.status
  prospector.scanner.resend_on_touch: true
  multiline:
    type: pattern
    pattern: '^APC'
    negate: true
    match: after
#    flush_pattern: '^END APC'
  processors:
    - add_fields:
        target: apc
        fields:
          source: apcupsd-status

And the log file is simple:

APC      : 001,037,0875
DATE     : 2021-11-28 15:12:21 -0800
HOSTNAME : apc-pi
VERSION  : 3.14.14 (31 May 2016) debian
UPSNAME  : APC1500
CABLE    : USB Cable
DRIVER   : USB UPS Driver
UPSMODE  :
STARTTIME: 2021-11-28 14:07:41 -0800
SHARE    :
MODEL    : Back-UPS RS 1500G
STATUS   : ONLINE
LINEV    : 121.0 Volts
LOADPCT  : 17.0 Percent
BCHARGE  : 100.0 Percent
TIMELEFT : 114.8 Minutes
MBATTCHG : -1 Percent
MINTIMEL : -1 Minutes
MAXTIME  : 0 Seconds
SENSE    : Medium
LOTRANS  : 88.0 Volts
HITRANS  : 147.0 Volts
ALARMDEL : No alarm
BATTV    : 27.2 Volts
LASTXFER : Unacceptable line voltage changes
NUMXFERS : 0
TONBATT  : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x05000008
SERIALNO : 3B1511X09577
BATTDATE : 2015-03-11
NOMINV   : 120 Volts
NOMBATTV : 24.0 Volts
NOMPOWER : 865 Watts
FIRMWARE : 865.L5 .D USB FW:L5
END APC  : 2021-11-28 15:12:22 -0800

No matter how many permutations I go through, I can't get a multiline message to go through though...

[Rios]

Are you using Logstash? If you do, add (?m) at the beginning.

grok {
    match => ["message", "(?m)%{LOGLEVEL:log_level}"]
}

[DPattee]

No, this is filebeat to Elasticsearch, this isn’t using logstash.

[DPattee]

Anyone else have thoughts on getting multiline filebeat injest working?

[stephenb (Stephen Brown)]

@DPattee Thanks... you helped me learn something today

You are using the new filestream input so the syntax is a bit different notice the parse section etc.
See Here you were using the old log input syntax (took me a while to figure that out!)

This should worked for me...

# filestream is an input for collecting log messages from files.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/filebeat-multiline/apc.log
    #- c:\programdata\elasticsearch\logs\*
  parsers:
    - multiline:
        type: pattern
        pattern: '^APC'
        negate: true
        match: after

I also built you out a little ingest pipeline to parse it up so

So set the pipeline

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  pipeline : discuss-apc

And Here is the pipeline , you can remove the other filebeat fields that you do want too.

PUT _ingest/pipeline/discuss-apc
{
  "processors": [
    {
      "kv": {
        "field": "message",
        "field_split": "\\n",
        "value_split": ":",
        "trim_key" : " ",
        "trim_value" : " "
      }
    },
    {
      "remove": {
        "field": "message"
      }
    }
  ]
}

And a result

GET filebeat-7.16.0-2021.12.10-000001/_search

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.16.0-2021.12.10-000001",
        "_type" : "_doc",
        "_id" : "hBn5pX0BkMxR-1fZmE6O",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "discuss"
          },
          "agent" : {
            "hostname" : "hyperion",
            "name" : "hyperion",
            "id" : "3430f760-613c-4756-8cf1-91694722d191",
            "type" : "filebeat",
            "ephemeral_id" : "f397ebd9-9fe9-4e13-9839-c2c91122f305",
            "version" : "7.16.0"
          },
          "ALARMDEL" : "No alarm",
          "DRIVER" : "USB UPS Driver",
          "UPSNAME" : "APC1500-1",
          "LASTXFER" : "Unacceptable line voltage changes",
          "TIMELEFT" : "114.8 Minutes",
          "LOTRANS" : "88.0 Volts",
          "SHARE" : "",
          "UPSMODE" : "",
          "MINTIMEL" : "-1 Minutes",
          "STATUS" : "ONLINE",
          "ecs" : {
            "version" : "1.12.0"
          },
          "SERIALNO" : "3B1511X09577",
          "XOFFBATT" : "N/A",
          "host" : {
            "hostname" : "hyperion",
            "os" : {
              "build" : "20G224",
              "kernel" : "20.6.0",
              "name" : "Mac OS X",
              "type" : "macos",
              "family" : "darwin",
              "version" : "10.16",
              "platform" : "darwin"
            },
            "ip" : [
              "fe80::aede:48ff:fe00:1122",
              "fe80::18d9:1a40:b8b2:bea0",
              "192.168.2.107",
              "fe80::d56c:87be:9b2:383d",
              "fe80::39ce:303b:f110:c7ac"
            ],
            "name" : "hyperion",
            "id" : "9E46F076-B7F1-53AA-921B-C2F983746B79",
            "mac" : [
              "ac:de:48:00:11:22",
              "7e:52:30:9c:ef:e0",
              "5c:52:30:9c:ef:e0",
              "8e:50:33:07:fa:e5",
              "82:b2:58:49:30:05",
              "82:b2:58:49:30:04",
              "82:b2:58:49:30:01",
              "82:b2:58:49:30:00",
              "a0:ce:c8:51:95:38",
              "82:b2:58:49:30:01"
            ],
            "architecture" : "x86_64"
          },
          "NUMXFERS" : "0",
          "CUMONBATT" : "0 Seconds",
          "SELFTEST" : "NO",
          "NOMPOWER" : "865 Watts",
          "input" : {
            "type" : "filestream"
          },

Finally you will want to create a template with mappings to you can set the data types for your fields.

[DPattee]

Ohhh I was using random old tutorials from the internet and didn't even realizing I was mixing two different things!

I'll give this a shot tonight, thanks