❗ Multiline Pattern Not Working in Filebeat 9.0.1 (Filestream Type Input)

[Ravi_Chandran]

Summary

Despite attempting multiple valid multiline.pattern configurations, Filebeat v9.0.1 fails to parse multiline log entries correctly from a plain text log file located inside a container. The logs contain Python stack traces and standard application logs. Filebeat continues to treat each line as a separate log event, breaking multiline messages across documents.

Environment Details

• Filebeat Version: 9.0.1
• Input Type: filestream
• Deployment: Docker (running inside a container)
• Log Source: Application writes to /app/log/techsavvyrc.log
• Multiline Log Format: Python stack traces + timestamped log entries

Sample Log File (techsavvyrc.log)

    self.raise_routing_exception(req)
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 500, in raise_routing_exception
    raise request.routing_exception  # type: ignore[misc]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/ctx.py", line 362, in match_request
    result = self.url_adapter.match(return_rule=True)  # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/werkzeug/routing/map.py", line 629, in match
    raise NotFound() from None
werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
2025-06-06 18:07:41,725 [INFO] service=techsavvyrc trace_id=fc54dfdec4be2a556bf7c8da0ed54a16 span_id=0d8069e49e6f7a83 Home page accessed
2025-06-06 18:08:12,803 [ERROR] service=techsavvyrc trace_id=2cf92917d8e8d5afe7846b2929e86277 span_id=92afc27e7bedd256 Unhandled exception: 405 Method Not Allowed: The method is not allowed for the requested URL.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 891, in dispatch_request
    self.raise_routing_exception(req)
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 500, in raise_routing_exception
    raise request.routing_exception  # type: ignore[misc]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/ctx.py", line 362, in match_request
    result = self.url_adapter.match(return_rule=True)  # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/werkzeug/routing/map.py", line 624, in match
    raise MethodNotAllowed(valid_methods=list(e.have_match_for)) from None
werkzeug.exceptions.MethodNotAllowed: 405 Method Not Allowed: The method is not allowed for the requested URL.

filebeat.yml Configuration

############################# General Filebeat Settings ##############################

name: "filebeat-techsavvyrc"
tags: ["techsavvyrc-app", "docker", "training", "testing"]
fields:
  environment: "testing"

############################### Paths & Idempotence ###############################

# Path to the directory where this filebeat.yml is located:
path.config: "/usr/share/filebeat"

# Path where Filebeat will store its registry (offsets, state, etc.)
# (Defaults to $path.home/data, but we override explicitly here.)
path.data: "/usr/share/filebeat/data"

# Path where Filebeat writes its own logs.
path.logs: "/usr/share/filebeat/logs_filebeat"

########################## Filebeat Input: filestream ##############################

filebeat.inputs:
  - type: filestream
    id: "techsavvyrc-app-log"
    enabled: true
    paths:
      - "/usr/share/filebeat/logs/techsavvyrc.log"
    multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} '
      #pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[[A-Z]+\]'
      #pattern: '^(\d{4}-\d{2}-\d{2})|(\s+File)|(\s+Traceback)'
      #pattern: '^\s'
      #pattern: '^\['
      match: after
      negate: false
      max_lines: 500
      timeout: 10s
      ignore_older: 0s
    processors:
      - add_fields:
          target: ""
          fields:
            service: "techsavvyrc-app"
            log_source: "internal"
      - add_host_metadata: {}

########################### Placeholders for Future Modules ########################

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

############################## Output Configuration #################################

#output.logstash:
#  hosts: ["logstash:5044"]
#  loadbalance: false
#  bulk_max_size: 200
#  worker: 1

output.console:
  enabled: true
  pretty: true
  
########################### Buffering & Queue Controls ##############################

queue.mem:
  events: 4096           
  flush.min_events: 1024 
  flush.timeout: 1s      

############################ Filebeat’s Own Logging #################################

logging.level: debug
logging.selectors: ["*multiline*"]
logging.to_files: true
logging.files:
  path: "/usr/share/filebeat/logs_filebeat"
  name: "filebeat.log"
  rotateeverybytes: 10485760
  keepfiles: 7
  permissions: 0644
logging.to_syslog: false
logging.json: false
logging.metrics.enabled: false
logging.metrics.period: 30s

Docker Compose Snippet

  filebeat:
    image: docker.elastic.co/beats/filebeat:${ELK_VERSION}
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./techsavvyrc/app/log:/usr/share/filebeat/logs:ro 
    depends_on:
      - logstash
    networks:
      - observability

Multiline Patterns Tried (All Failed)

We tested all the following patterns with appropriate negate and match logic, but Filebeat still failed to group multiline events correctly:

1) pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} '
2) pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[[A-Z]+\]'
3) pattern: '^(\d{4}-\d{2}-\d{2})|(\s+File)|(\s+Traceback)'
4) pattern: '^\s'
5) pattern: '^\['

Observed Filebeat Console Output

{
  "@timestamp": "2025-06-08T01:03:12.929Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "host": {
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ]
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": "2025-06-08 00:47:02,070 [INFO] service=techsavvyrc trace_id=0f0c92999ee3fb4d4cbd675af83568aa span_id=4f1e9ca612d79253 Home page accessed",
  "log_source": "internal",
  "agent": {
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513"
  },
  "log": {
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    },
    "offset": 57514
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app"
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "message": "2025-06-08 01:04:06,129 [INFO] service=techsavvyrc trace_id=1b81f97de2fcee75787bc37f36608354 span_id=3775fdb29dda10dc Home page accessed",
  "input": {
    "type": "filestream"
  },
  "service": "techsavvyrc-app",
  "host": {
    "architecture": "x86_64",
    "os": {
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  },
  "log": {
    "offset": 57651,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "log_source": "internal"
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "message": "2025-06-08 01:04:06,657 [ERROR] service=techsavvyrc trace_id=d3ac98f8ebf989fb4d7acd2b39139c98 span_id=71ae04aba190dd39 Unhandled exception: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "input": {
    "type": "filestream"
  },
  "log_source": "internal",
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945"
  },
  "log": {
    "offset": 57788,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "service": "techsavvyrc-app",
  "host": {
    "architecture": "x86_64",
    "os": {
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    },
    "offset": 58064
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "host": {
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false
  },
  "message": "Traceback (most recent call last):",
  "input": {
    "type": "filestream"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "fields": {
    "environment": "testing"
  },
  "host": {
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d"
  },
  "agent": {
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1"
  },
  "message": "  File \"/usr/local/lib/python3.11/site-packages/flask/app.py\", line 917, in full_dispatch_request",
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "ecs": {
    "version": "8.0.0"
  },
  "log": {
    "offset": 58099,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": "    rv = self.dispatch_request()",
  "log": {
    "offset": 58197,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "log_source": "internal",
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  },
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "host": {
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  },
  "log": {
    "offset": 58230,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "message": "         ^^^^^^^^^^^^^^^^^^^^^^^",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "host": {
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ]
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "offset": 58263,
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    }
  },
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "host": {
    "architecture": "x86_64",
    "os": {
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": "  File \"/usr/local/lib/python3.11/site-packages/flask/app.py\", line 891, in dispatch_request",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "agent": {
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": "    self.raise_routing_exception(req)",
  "service": "techsavvyrc-app",
  "agent": {
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513"
  },
  "log": {
    "offset": 58356,
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "log_source": "internal",
  "host": {
    "os": {
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "offset": 58394,
    "file": {
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000"
    }
  },
  "message": "  File \"/usr/local/lib/python3.11/site-packages/flask/app.py\", line 500, in raise_routing_exception",
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "host": {
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d"
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "log_source": "internal",
  "agent": {
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513"
  },
  "ecs": {
    "version": "8.0.0"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "offset": 58494,
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": "    raise request.routing_exception  # type: ignore[misc]",
  "input": {
    "type": "filestream"
  },
  "host": {
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow"
    }
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "offset": 58552,
    "file": {
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "host": {
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow"
    }
  },
  "message": "    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^",
  "fields": {
    "environment": "testing"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "input": {
    "type": "filestream"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "agent": {
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": "  File \"/usr/local/lib/python3.11/site-packages/flask/ctx.py\", line 362, in match_request",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "host": {
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ]
  },
  "log": {
    "offset": 58588,
    "file": {
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log"
    }
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "log": {
    "offset": 58678,
    "file": {
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log"
    }
  },
  "input": {
    "type": "filestream"
  },
  "log_source": "internal",
  "host": {
    "os": {
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64"
  },
  "agent": {
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1"
  },
  "message": "    result = self.url_adapter.match(return_rule=True)  # type: ignore",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app"
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "ecs": {
    "version": "8.0.0"
  },
  "log": {
    "offset": 58748,
    "file": {
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000"
    }
  },
  "message": "             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^",
  "log_source": "internal",
  "host": {
    "os": {
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64"
  },
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  }
}

Steps Tried

• Verified the log file is accessible inside the container.
• Confirmed format: each exception stack trace is indented, and new entries start with a clear timestamp.
• Cleared registry:
  rm -rf /usr/share/filebeat/data/registry
• Enabled debug:
  filebeat -e -d "multiline"

Expected Behavior

Filebeat should group stack traces and related lines with the original timestamped log line as one cohesive event.

Actual Behavior

Each line, including indented traceback lines, is sent as a separate log event. No multiline grouping occurs.

---

Request for Help

Please assist with:

• Identifying if this is a regression or behavioral change in Filebeat v9.0.1.
• Confirming whether the registry, encoding, or buffering logic changed.
• Suggesting a working multiline pattern or workaround.

Docker Commands Used

1)  docker run --rm -it -v "$(pwd)/filebeat/filebeat.yml":/usr/share/filebeat/filebeat.yml -v "$(pwd)/techsavvyrc/app/log":/usr/share/filebeat/logs docker.elastic.co/beats/filebeat:9.0.1 -e -d "multiline"
2)  docker run --rm -it -v "$(pwd)/filebeat/filebeat.yml":/usr/share/filebeat/filebeat.yml -v "$(pwd)/techsavvyrc/app/log":/usr/share/filebeat/logs docker.elastic.co/beats/filebeat:9.0.1 bash
3)  docker logs -f filebeat
4)  docker exec -it filebeat /bin/bash

[stephenb (Stephen Brown)]

Hi @Ravi_Chandran Welxomne to the community ....

You are using the wrong syntax

@Ravi_Chandran

filebeat.inputs:
  - type: filestream <<< FILESTREAM 
    id: "techsavvyrc-app-log"
    enabled: true
    paths:
      - "/usr/share/filebeat/logs/techsavvyrc.log"
    multiline: <<<< WRONG SYNTAX uses parsers sytax 
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} '
      #pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[[A-Z]+\]'
      #pattern: '^(\d{4}-\d{2}-\d{2})|(\s+File)|(\s+Traceback)'
      #pattern: '^\s'
      #pattern: '^\['
      match: after
      negate: false << Probably should be true
      max_lines: 500
      timeout: 10s
      ignore_older: 0s

See here should use the parsers syntax

Please note that the example below only works with filestream input, and not with log input.

parsers:
- multiline:
    type: pattern
    pattern: '^\['
    negate: true
    match: after

Your is probably something like

parsers:
- multiline:
    type: pattern
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} '
    negate: true
    match: after

[Ravi_Chandran]

Hi @Stephenb,

Thank you so much for the quick response — your suggestion worked like a charm and was truly a saviour!

I had been struggling with various multiline patterns under filestream without success, but placing the configuration under the parsers: section, as you recommended, immediately resolved the issue. This saved me a lot of time and confusion.

For others facing similar challenges, I’m sharing a working setup below using Filebeat 9.0.1, Logstash, and Docker Compose, along with a link to my GitHub repository that contains all supporting files used in my local observability test setup.

⚠️ Disclaimer:
This setup is intended only for testing and learning purposes. It lacks security configurations (such as TLS, authentication, access control, etc.) and must not be used in a production environment without proper hardening.

Working Setup

docker-compose.yml

volumes:
  es_data:
  portainer_data:

networks:
  observability:
    driver: bridge

services:

  techsavvyrc-app:
    build: ./techsavvyrc
    container_name: techsavvyrc-app
    ports:
      - 8000:8000
    volumes:
      - ./techsavvyrc/app/log:/app/log
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
    depends_on:
      - otel-collector
      - apm-server
      - prometheus
    logging:
      driver: "json-file"
    networks:
      - observability

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
    container_name: elasticsearch
    ports:
      - 9200:9200
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    volumes:
      - es_data:/usr/share/elasticsearch/data
    networks:
      - observability

  kibana:
    image: docker.elastic.co/kibana/kibana:${ELK_VERSION}
    container_name: kibana
    ports:
      - 5601:5601
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - observability

  filebeat:
    image: docker.elastic.co/beats/filebeat:${ELK_VERSION}
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./techsavvyrc/app/log:/usr/share/filebeat/logs:ro
    depends_on:
      - logstash
    networks:
      - observability

  logstash:
    image: docker.elastic.co/logstash/logstash:${ELK_VERSION}
    container_name: logstash
    volumes:
      - ./logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
      - ./logstash/logstash.yml:/usr/share/logstash/config/logstash.yml
    ports:
      - 5044:5044
    depends_on:
      - elasticsearch
    networks:
      - observability
    environment:
      - LS_JAVA_OPTS=-Xms2g -Xmx2g

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports:
      - 9090:9090
    networks:
      - observability

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    networks:
      - observability

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    volumes:
      - ./otel/otel-config.yaml:/etc/otelcol/config.yaml:ro
    command: ["--config=/etc/otelcol/config.yaml"]
    ports:
      - "4318:4318"
      - "8888:8888"
      - "8889:8889"
    networks:
      - observability

  portainer:
    image: portainer/portainer-ce
    container_name: portainer
    ports:
      - "9000:9000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    restart: unless-stopped
    networks:
      - observability

filebeat.yml

# ===================================================================================
# Filebeat Configuration File - Version 9.0.1
#
# Description:
#   This configuration is designed specifically for a containerized observability setup.
#   It captures both application logs and Docker container logs using Filebeat inside
#   a Docker environment.
#
# Purpose:
#   - Educational and training use only.
#   - Not suitable for production environments due to the absence of essential security
#      features like TLS encryption, user authentication, and hardened logging.
#
# GitHub Reference:
#   Full stack and related files are available here:
#   https://github.com/TechSavvyRC/techsavvyrc-demo
#
# Official Documentation:
#   https://www.elastic.co/guide/en/beats/filebeat/9.0/filebeat-reference-yml.html
# ===================================================================================

# ================================= General Settings ================================
# Basic identification and tagging for this Filebeat instance.
# Sets the name, adds tags for filtering, and includes custom fields in all events.
name: "filebeat-techsavvyrc"                                # Custom name for this Filebeat instance
tags: ["techsavvyrc-app", "docker", "training", "testing"]  # Tags added to every event
fields:                                                     # Additional fields in each event
  environment: "testing"                                    # Custom fields added to all events


# ================================== Path Settings ==================================
# Directory paths where Filebeat stores configuration, data, and its own log files.
# These paths ensure proper file organization within the container.
path.config: "/usr/share/filebeat"                     # Base path for config
path.data: "/usr/share/filebeat/data"                  # Path to registry/state
path.logs: "/usr/share/filebeat/logs_filebeat"         # Path for internal logs


# =================================== Input Settings ================================
# Defines input sources to harvest logs from files, containers, or other systems.
# Each input can include parsing rules (e.g., multiline handling) and preprocessing
# logic (e.g., enrichment via processors).
# Supported input types include: filestream, docker, journald, tcp/udp, etc.
filebeat.inputs:

  # Input to monitor application log file from a mounted path inside the container.
  # Includes multiline parsing for stack traces and adds custom metadata to events.
  - type: filestream                                    # Recommended in Filebeat 9.x
    id: "techsavvyrc-app-log"                           # Unique ID for the input
    enabled: true                                       # Enable this input
    paths:
      - "/usr/share/filebeat/logs/techsavvyrc-app.log"  # Mounted log file path
    # Multiline parsing for Python stack traces
    parsers:
      - multiline:
          type: pattern
          pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} '  # Timestamp pattern
          match: after                                  # Append continuation lines *after* matched lines
          negate: true                                  # Treat non-matching lines as continuation
          max_lines: 500                                # Max lines per event
          timeout: 10s                                  # Flush if no new line within 10 seconds
          #ignore_older: 0s                             # Do not skip old files
    # Event enrichment
    processors:
      - add_fields:                                     # Add custom fields to events
          target: ""
          fields:
            service: "techsavvyrc-app"
            log_source: "internal"
      - add_host_metadata: {}                           # Adds host metadata (OS, hostname, etc.)

  # Streams logs from all Docker containers (mounted via /var/lib/docker/containers)
  # Docker log files are JSON formatted. We decode JSON here.
  - type: filestream
    id: "docker-containers"
    enabled: true
    paths:
      - /var/lib/docker/containers/*/*.log              # All container logs
    symlinks: true                                      # Follow symlinks (just in case)
    parsers:
      - ndjson:
          keys_under_root: true                         # Place JSON keys at root level
          overwrite_keys: true                          # Overwrite conflicting keys like "message"
    processors:
      - add_fields:
          target: ""
          fields:
            service: "docker-container"                 # Mark logs as from docker container
            log_source: "docker"
      - add_docker_metadata:                            # Adds container ID, name, labels, etc.
          host: "unix:///var/run/docker.sock"
          match_source: true


# ================================== Module Settings ================================
# Configuration for Filebeat modules (nginx, system, etc.).
# Currently disabled but ready for future use - modules provide pre-built configs for common services.
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false       # Enable for dynamic module loading
  reload.period: 10s          # How often to check for module file changes


# ================================= Output Settings =================================
# Configuration for where to send processed log events.
# Primary output is Logstash, with alternative outputs commented for easy activation.
# Primary output to Logstash
#output.logstash:
#  hosts: ["logstash:5044"]          # Logstash listening on port 5044
#  loadbalance: false                # Set true for multi-node logstash
#  bulk_max_size: 200                # Max events per bulk request
#  worker: 1                         # Number of concurrent workers

# Direct output to Elasticsearch, uncomment if used without Logstash.
# output.elasticsearch:
#   hosts: ["http://elasticsearch:9200"]              # ES URL
#   index: "techsavvyrc-%{+yyyy.MM.dd}"               # Dynamic index name
#   pipeline: "filebeat-%{[agent.version]}-pipeline"  # Optional ingest pipeline
#   username: "elastic"                               # Auth user
#   password: "changeme"                              # Auth password
#   ssl.enabled: false                                # Enable for HTTPS
#   bulk_max_size: 200
#   worker: 1
output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  indices:
    - index: "techsavvyrc-app-%{+yyyy.MM.dd}"     # For app logs
      when.equals:
        fields.service: "techsavvyrc-app"
    - index: "docker-containers-%{+yyyy.MM.dd}"   # For container logs
      when.equals:
        fields.service: "docker-containers"

# Console output for debugging:
#output.console:
#  enabled: true
#  pretty: true

# ================================= Queue Settings ==================================
# In-memory queue configuration for buffering events before output.
# Controls memory usage and batching behavior for optimal performance.
queue.mem:
  events: 4096                # maximum events to keep in memory (default ~4096)
  flush.min_events: 1024      # force a flush to output when ≥1024 events are ready
  flush.timeout: 1s           # force a flush if no new events after 1s


# =============================== Dashboard Setup ===============================
# Kibana dashboard and index template setup configuration.
# When enabled, automatically creates dashboards and templates during setup phase.
# Kibana dashboard configuration (uncomment to enable)
# setup.dashboards.enabled: true
# setup.dashboards.index: ".filebeat"         # where dashboards are stored in ES
# setup.dashboards.overwrite: false           # set to true to re-import every setup run
# setup.dashboards.directory: "/path/to/your/custom/dashboards"

# setup.kibana:
#   hosts: ["http://kibana:5601"]           # Kibana endpoint
#   username: "kibana_system"               # Optional authentication
#   password: "changeme"


# ================================ Logging Settings ===============================
# Configuration for Filebeat's own internal logging.
# Controls log levels, file rotation, and where Filebeat writes its operational logs.
logging.level: debug                          # Options: debug, info, warning, error, critical
logging.selectors: ["*multiline*"]            # Focus on multiline logs
logging.to_files: true
logging.files:
  path: "/usr/share/filebeat/logs_filebeat"   # Log file location
  name: "filebeat.log"
  rotateeverybytes: 10485760                  # Rotate every 10 MB
  keepfiles: 7                                # Keep last 7 files before deletion
  permissions: 0644
logging.to_syslog: false
logging.json: false
logging.metrics.enabled: false
logging.metrics.period: 30s


# ============================== Monitoring Settings ==============================
# Self-monitoring configuration for Filebeat performance metrics.
# When enabled, sends Filebeat's own operational metrics to Elasticsearch for monitoring.
monitoring.enabled: false

# To send to Elasticsearch:
# monitoring.elasticsearch:
#   hosts: ["http://elasticsearch:9200"]
# To register monitoring dashboards in Kibana:
# monitoring.kibana:
#   hosts: ["http://kibana:5601"]
#   username: "kibana_system"
#   password: "changeme"
# If you have Elastic Agent, disable Filebeat’s built-in monitoring:
# monitoring.elastic_agent.enabled: true

My Git Repo: TechSavvyRC GIT Repo