Empty lines in multiline pattern(python error traceback) in filebeat input are not getting parsed correctly?

The log line which should be harvested and published to logstash as a single line:

[pid: 17318|app: 0|req: 1/2] 10.14.206.28 (jaavedkhan) {60 vars in 1296 bytes} [Mon Dec 30 15:51:38 2019] GET /en/ => generated 27 bytes in 711 msecs (HTTP/1.1 500) 6 headers in 316 bytes (1 switches on core 0)
Mon Dec 30 15:51:39 2019 - announcing my loyalty to the Emperor...


Internal Server Error: /en/
Traceback (most recent call last):
  File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "./core/views.py", line 31, in get
    1/0
ZeroDivisionError: division by zero

my filebeat configurations:

filebeat:
  inputs:
    - type: log
      paths:
        - "/var/log/uwsgi/vassals/dsr-incentives.log"
      fields_under_root: true
      multiline:
        pattern: '\[pid:\s*\d*\|app:'
        negate: true
        match: after
      fields:
        log_type: app-access
        appserver: uwsgi
        app: dsr-incentives
        server_name: server-name.domain.com

I checked the multiline pattern https://play.golang.org with the log line:

enter image description here The result is as expected but the harvester is splitting the log line at "Internat server error"

    Publish event: {
  "@timestamp": "2019-12-30T13:02:56.564Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.1"
  },
  "log": {
    "offset": 128736,
    "file": {
      "path": "/var/log/uwsgi/vassals/dsr-incentives.log"
    },
    "flags": [
      "multiline"
    ]
  },
  "appserver": "uwsgi",
  "server_name": "xyz",
  "log_type": "app-access",
  "host": {
    "name": "xyz"
  },
  "agent": {
    "hostname": "apps-1",
    "id": "d3417bc3-213c-4d5e-a9b5-2273178262d0",
    "version": "7.5.1",
    "name": "xyz",
    "type": "filebeat",
    "ephemeral_id": "125578d6-44d1-4103-94bc-a1d062091487"
  },
  "message": "Internal Server Error: /en/\nTraceback (most recent call last):\n  File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/exception.py\", line 34, in inner\n    response = get_response(request)\n  File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/base.py\", line 126, in _get_response\n    response = self.process_exception_by_middleware(e, request)\n  File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/base.py\", line 124, in _get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n  File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/views/generic/base.py\", line 68, in view\n    return self.dispatch(request, *args, **kwargs)\n  File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/views/generic/base.py\", line 88, in dispatch\n    return handler(request, *args, **kwargs)\n  File \"./core/views.py\", line 31, in get\n    1/0\nZeroDivisionError: division by zero",
  "tags": [
    "filebeat"
  ],
  "input": {
    "type": "log"
  },
  "app": "dsr-incentives",
  "ecs": {
    "version": "1.1.0"
  }
}

I think the problem is that the multiline is getting split at empty lines that are appearing in logs before "Internal Server Error".

Update The log message is getting parsed correctly when there are no empty lines above "Internal Server Error" line

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.