# Filebeat multiline config not working

**URL:** <https://discuss.elastic.co/t/filebeat-multiline-config-not-working/290383>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 28, 2021, 11:14pm UTC](https://discuss.elastic.co/t/filebeat-multiline-config-not-working/290383 "2021-11-28T23:14:42Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![DPattee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dpattee/32/37772_2.png) [@DPattee](https://discuss.elastic.co/u/DPattee)\
**Post date:** [November 28, 2021, 11:14pm UTC](https://discuss.elastic.co/t/filebeat-multiline-config-not-working/290383/1 "2021-11-28T23:14:42Z")

</div>

I have a 3rd party app that spits out a text file with multiple lines for a single event. An event has a consistent start line and an end line.

I have tried filebeat configurations that grab everything after a specific regex, everything before a specific regex, and everything between two regexes.

I've toggled 'negate' off and on and switched the main pattern from my start to my end patterns.

I've changed my regex from being just a match at the start of a line, to being that start of the line match plus a .\* to grab the entire line.

None of these changes "do anything" - I still end up with every line of the file as a distinct record in elastic.

My yaml is valid, I double checked indentation with yamllint and filebeat is loading and running everything else (including the 'add field' processor immediately below the multiline part that it seems to be ignoring.

Even though I'm using some of the predefined modules for other files, I've just defined these two particular log patterns directly in my filebeat.yml. The relevant top portion (beyond this is just the kibana/elastic setup portions)

```auto
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 300s

# Extra inputs
filebeat.inputs:
- type: filestream
  paths:
    - /var/log/apcupsd*.events
  processors:
    - add_fields:
        target: apc
        fields:
          source: apcupsd-event
- type: filestream
  paths:
    - /var/log/apcupsd*.status
  prospector.scanner.resend_on_touch: true
  multiline:
    type: pattern
    pattern: '^APC'
    negate: true
    match: after
# flush_pattern: '^END APC'
  processors:
    - add_fields:
        target: apc
        fields:
          source: apcupsd-status

```

And the log file is simple:

```auto
APC : 001,037,0875
DATE : 2021-11-28 15:12:21 -0800
HOSTNAME : apc-pi
VERSION : 3.14.14 (31 May 2016) debian
UPSNAME : APC1500
CABLE : USB Cable
DRIVER : USB UPS Driver
UPSMODE :
STARTTIME: 2021-11-28 14:07:41 -0800
SHARE :
MODEL : Back-UPS RS 1500G
STATUS : ONLINE
LINEV : 121.0 Volts
LOADPCT : 17.0 Percent
BCHARGE : 100.0 Percent
TIMELEFT : 114.8 Minutes
MBATTCHG : -1 Percent
MINTIMEL : -1 Minutes
MAXTIME : 0 Seconds
SENSE : Medium
LOTRANS : 88.0 Volts
HITRANS : 147.0 Volts
ALARMDEL : No alarm
BATTV : 27.2 Volts
LASTXFER : Unacceptable line voltage changes
NUMXFERS : 0
TONBATT : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x05000008
SERIALNO : 3B1511X09577
BATTDATE : 2015-03-11
NOMINV : 120 Volts
NOMBATTV : 24.0 Volts
NOMPOWER : 865 Watts
FIRMWARE : 865.L5 .D USB FW:L5
END APC : 2021-11-28 15:12:22 -0800

```

No matter how many permutations I go through, I can't get a multiline message to go through though...

 ![Screen Shot 2021-11-28 at 3.14.03 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8b76a41195b15e30dea90f732d8513f7cb6ba979.png)

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [November 29, 2021, 6:40am UTC](https://discuss.elastic.co/t/filebeat-multiline-config-not-working/290383/2 "2021-11-29T06:40:11Z")

</div>

Are you using Logstash? If you do, add (?m) at the beginning.

```auto
grok {
    match => ["message", "(?m)%{LOGLEVEL:log_level}"]
}

```

---

<div class="post-metadata">

**Author:** ![DPattee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dpattee/32/37772_2.png) [@DPattee](https://discuss.elastic.co/u/DPattee)\
**Post date:** [November 29, 2021, 3:45pm UTC](https://discuss.elastic.co/t/filebeat-multiline-config-not-working/290383/3 "2021-11-29T15:45:58Z")

</div>

No, this is filebeat to Elasticsearch, this isn’t using logstash.

---

<div class="post-metadata">

**Author:** ![DPattee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dpattee/32/37772_2.png) [@DPattee](https://discuss.elastic.co/u/DPattee)\
**Post date:** [December 10, 2021, 6:41pm UTC](https://discuss.elastic.co/t/filebeat-multiline-config-not-working/290383/4 "2021-12-10T18:41:05Z")

</div>

Anyone else have thoughts on getting multiline filebeat injest working?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 10, 2021, 8:13pm UTC](https://discuss.elastic.co/t/filebeat-multiline-config-not-working/290383/5 "2021-12-10T20:13:15Z")

</div>

@DPattee Thanks... you helped me learn something today

You are using the new `filestream` input so the syntax is a bit different notice the parse section etc.  
See [Here](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_parsers) you were using the old `log` input syntax (took me a while to figure that out!)

This should worked for me...

```auto
# filestream is an input for collecting log messages from files.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/filebeat-multiline/apc.log
    #- c:\programdata\elasticsearch\logs\*
  parsers:
    - multiline:
        type: pattern
        pattern: '^APC'
        negate: true
        match: after

```

I also built you out a little ingest pipeline to parse it up so

So set the pipeline

```auto
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  pipeline : discuss-apc

```

And Here is the pipeline , you can remove the other filebeat fields that you do want too.

```auto
PUT _ingest/pipeline/discuss-apc
{
  "processors": [
    {
      "kv": {
        "field": "message",
        "field_split": "\\n",
        "value_split": ":",
        "trim_key" : " ",
        "trim_value" : " "
      }
    },
    {
      "remove": {
        "field": "message"
      }
    }
  ]
}

```

And a result

```auto
GET filebeat-7.16.0-2021.12.10-000001/_search

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.16.0-2021.12.10-000001",
        "_type" : "_doc",
        "_id" : "hBn5pX0BkMxR-1fZmE6O",
        "_score" : 1.0,
        "_source" : {
          "container" : {
            "id" : "discuss"
          },
          "agent" : {
            "hostname" : "hyperion",
            "name" : "hyperion",
            "id" : "3430f760-613c-4756-8cf1-91694722d191",
            "type" : "filebeat",
            "ephemeral_id" : "f397ebd9-9fe9-4e13-9839-c2c91122f305",
            "version" : "7.16.0"
          },
          "ALARMDEL" : "No alarm",
          "DRIVER" : "USB UPS Driver",
          "UPSNAME" : "APC1500-1",
          "LASTXFER" : "Unacceptable line voltage changes",
          "TIMELEFT" : "114.8 Minutes",
          "LOTRANS" : "88.0 Volts",
          "SHARE" : "",
          "UPSMODE" : "",
          "MINTIMEL" : "-1 Minutes",
          "STATUS" : "ONLINE",
          "ecs" : {
            "version" : "1.12.0"
          },
          "SERIALNO" : "3B1511X09577",
          "XOFFBATT" : "N/A",
          "host" : {
            "hostname" : "hyperion",
            "os" : {
              "build" : "20G224",
              "kernel" : "20.6.0",
              "name" : "Mac OS X",
              "type" : "macos",
              "family" : "darwin",
              "version" : "10.16",
              "platform" : "darwin"
            },
            "ip" : [
              "fe80::aede:48ff:fe00:1122",
              "fe80::18d9:1a40:b8b2:bea0",
              "192.168.2.107",
              "fe80::d56c:87be:9b2:383d",
              "fe80::39ce:303b:f110:c7ac"
            ],
            "name" : "hyperion",
            "id" : "9E46F076-B7F1-53AA-921B-C2F983746B79",
            "mac" : [
              "ac:de:48:00:11:22",
              "7e:52:30:9c:ef:e0",
              "5c:52:30:9c:ef:e0",
              "8e:50:33:07:fa:e5",
              "82:b2:58:49:30:05",
              "82:b2:58:49:30:04",
              "82:b2:58:49:30:01",
              "82:b2:58:49:30:00",
              "a0:ce:c8:51:95:38",
              "82:b2:58:49:30:01"
            ],
            "architecture" : "x86_64"
          },
          "NUMXFERS" : "0",
          "CUMONBATT" : "0 Seconds",
          "SELFTEST" : "NO",
          "NOMPOWER" : "865 Watts",
          "input" : {
            "type" : "filestream"
          },

```

Finally you will want to create a template with mappings to you can set the data types for your fields.

---

<div class="post-metadata">

**Author:** ![DPattee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dpattee/32/37772_2.png) [@DPattee](https://discuss.elastic.co/u/DPattee)\
**Post date:** [December 13, 2021, 10:40pm UTC](https://discuss.elastic.co/t/filebeat-multiline-config-not-working/290383/6 "2021-12-13T22:40:30Z")

</div>

Ohhh I was using random old tutorials from the internet and didn't even realizing I was mixing two different things!

I'll give this a shot tonight, thanks 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2022, 12:41am UTC](https://discuss.elastic.co/t/filebeat-multiline-config-not-working/290383/7 "2022-01-11T00:41:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.