Filebeat multi line pattern not working

Elastic Stack Beats
1

Following is the multiline pattern config we have used.

- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: testfile

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - 'F:\ProgramData\Secure Messaging\Logs\DEV_BETA\test.txt'
    #- c:\programdata\elasticsearch\logs\*
  multiline.pattern: '^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} (AM|PM)' 
  multiline.negate: true
  multiline.match: after
  scan.start_position: beginning
  ignore_older: 0s

following is the log file which we want parse it as single line

2025/04/28 06:17:04 AM
System.Exception: ServiceCode not found wrongservicecode. 
2025/04/29 06:17:05 AM
System.Exception: ServiceCode not found wrongservicecode.
2025/04/29 06:17:06 AM
System.Exception: ServiceCode not found wrongservicecode.
2025/04/29 06:17:07 AM
System.Exception: ServiceCode not found wrongservicecode.

Instead of 4 documents, above log is indexed as 8 seperate documents. Currently using 8.17.4 version

Can you point me what i am missing out.
Thanks in advance.
Mani.

2

The slash mark / must be \/. Can you test with this:
multiline.pattern: '^\d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} (AM|PM)'

3

Have configured my filebeat to write output to file for testing purpose and following is the output i get. Still it considers as seperate line instead of one.

{"@timestamp":"2025-05-19T14:08:38.902Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.17.4"},"agent":{"ephemeral_id":"ecd46137-7e40-4fd5-9fff-a77debefa760","id":"3c532dc1-1354-44fb-a59f-844b38c77e6f","name":"4932028d7987070","type":"filebeat","version":"8.17.4"},"ecs":{"version":"8.0.0"},"log":{"offset":191,"file":{"vol":"3595704835","path":"F:\\ProgramData\\Secure Messaging\\Logs\\DEV_BETA\\test.txt","idxhi":"131072","idxlo":"72208"}},"message":"System.Exception: ServiceCode not found wrongservicecode.","input":{"type":"filestream"},"fields":{"application":"test"},"host":{"ip":["",""],"mac":[""],"name":"","hostname":"","architecture":"x86_64","os":{"type":"windows","platform":"windows","version":"10.0","family":"windows","name":"Windows Server 2022 Datacenter","kernel":"10.0.20348.3451 (WinBuild.160101.0800)","build":"20348.3453"},"id":"939c207a-7ddc-4816-8cc8-2ff3f4fd7cc1"}}
{"@timestamp":"2025-05-19T14:08:38.902Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.17.4"},"ecs":{"version":"8.0.0"},"log":{"file":{"idxlo":"72208","vol":"3595704835","path":"F:\\ProgramData\\Secure Messaging\\Logs\\DEV_BETA\\test.txt","idxhi":"131072"},"offset":250},"message":"2025/04/29 06:17:07 AM","input":{"type":"filestream"},"fields":{"application":"test"},"host":{"architecture":"x86_64","os":{"kernel":"10.0.20348.3451 (WinBuild.160101.0800)","build":"20348.3453","type":"windows","platform":"windows","version":"10.0","family":"windows","name":"Windows Server 2022 Datacenter"},"id":"939c207a-7ddc-4816-8cc8-2ff3f4fd7cc1","ip":[""],"mac":[""],"hostname":"","name":""},"agent":{"version":"8.17.4","ephemeral_id":"ecd46137-7e40-4fd5-9fff-a77debefa760","id":"3c532dc1-1354-44fb-a59f-844b38c77e6f","name":"4932028d7987070","type":"filebeat"}}

when i used the regex and searched the log file using the regex pattern in notepad++, it rightly identified the text. Not sure why in filebeat its not working.

4

Because the file was read.
Please rename/remove filebeat\data\registry\filebeat\log.json

1 Like
5

I did that already and the result is same. i deleted the data folder and the result is same.

6

You are using the wrong syntax for filestream that is old log type syntax

- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/filebeat-multiline/test-log-378301.log
    #- c:\programdata\elasticsearch\logs\*
  parsers:
    - multiline:
        type: pattern
        pattern: '^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} (AM|PM)' 
        negate: true
        match: after

That worked for me results

{
	"@timestamp": "2025-05-19T17:03:54.690Z",
	"@metadata": {
		"beat": "filebeat",
		"type": "_doc",
		"version": "8.17.2"
	},
	"container": {
		"id": "discuss"
	},
	"message": "2025/04/29 06:17:07 AM\nSystem.Exception: ServiceCode not found wrongservicecode.\n",
	"log": {
		"flags": [
			"multiline"
		],
		"offset": 267,
		"file": {
			"path": "/Users/sbrown/workspace/sample-data/discuss/filebeat-multiline/test-log-378301.log",
			"device_id": "16777221",
			"inode": "144197757"
		}
	},
	"input": {
		"type": "filestream"
	},
7

thanks @stephenb it worked.

1 Like