FILEBEAT REGULAR EXPRESSION FAILING IN MULTI LINE PATTERN

Elastic Stack Beats
1

We are using multiline pattern for file beats to parse the application logs.
Multiline patterns is not able to merge related line as one log

PS : We are using https://play.golang.org/ for testing.
We are not able to find any regex for negative lookups

Regular Expression :

  1. ^[[:space:]]
  2. ^[[:space:]]+(at|.{3})\b|^Traceback (most recent call last):
  3. ^[[:space:]]|^Traceback|^exception

Multiline Config :
multiline.negate: false
multiline.match: after

Expected Output :
Matched Pattern 1:
ERROR 2019-04-18 12:04:11,193 publisher_viewsets 16 140543468234496 (2, 'XYZXYZ_ERR_RECORD_NOT_FOUND', 'src/XXXXXXXXXXXXXXX', 113)
Traceback (most recent call last):
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 131, in XYZXYZ
user_bins = XYZXYZ_ERR_RECORD_NOT_FOUND.XYZXYZ(XYZXYZ)
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 203, in XYZXYZ
raise e
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 199, in XYZXYZ
key, meta, bin = self.XYZ.get(key)
exception.RecordNotFound: (2, 'XYZXYZ_ERR_RECORD_NOT_FOUND', 'src/XYZ/XYZ/XYZ.c', 113)

Matched Pattern 2:
ERROR 2019-04-19 11:38:48,148 event_viewsets 16 140543468234496 Signature has expired
Traceback (most recent call last):
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 131, in XYZXYZ
user_bins = XYZXYZ_ERR_RECORD_NOT_FOUND.XYZXYZ(XYZXYZ)
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 203, in XYZXYZ
raise e
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 199, in XYZXYZ
key, meta, bin = self.XYZ.get(key)
File "/usr//opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 284, in verify
'signatures' + repr(self.verifylog))
xyz.xyz.XYZ: Verification failed for all XYZ['Failed: [Invalid("Verification failed {XYZ('Temporary Text.',)}",)]']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/XXXXXXXXX/XXXXXXXXX/XXXXXXXXX/views/event_viewsets.py", line 269, in track
client_cookie, is_new, _ = XYZ_or_XYZ_XYZ(request, UID_COOKIE_NAME)
File "/opt/XXXXXXXXX/XXXXXXXXX/XXXXXXXXX/utils.py", line 164, in XYZ_or_XYZ_XYZ
XYZ_or_XYZ_XYZ = ClientCookie.get_decoded_cookie(arbitrary_cookie).XYZ_or_XYZ_XYZ
File "/opt/XXXXXXXXX/XXXXXXXXX/XXXXXXXXX/utils.py", line 69, in get_decoded_cookie
payload = jwt_decode_handler(encoded_cookie)
File "/usr/XXXXXXXXX/XXXXXXXXX/XXXXXXXXX/api.xx", line 175, in _XYZ_exp
raise ExpiredSignatureError('XYZ has XYZ')
XYZ.XYZ.ExpiredSignatureError: XYZ XYZ XYZ

Matched Pattern 3:
DEBUG 2019-04-18 12:04:11,706 event_viewsets 17 140543334016768 Returning response with client cookie:XYZ

Matched Pattern 4:
WARNING 2019-04-18 10:39:26,416 base 19 140544438826752 Not Found: /XYZ/XYZ.exp

Sample Log :
\
ERROR 2019-04-18 12:04:11,193 publisher_viewsets 16 140543468234496 (2, 'XYZXYZ_ERR_RECORD_NOT_FOUND', 'src/XXXXXXXXXXXXXXX', 113)
Traceback (most recent call last):
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 131, in XYZXYZ
user_bins = XYZXYZ_ERR_RECORD_NOT_FOUND.XYZXYZ(XYZXYZ)
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 203, in XYZXYZ
raise e
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 199, in XYZXYZ
key, meta, bin = self.XYZ.get(key)
exception.RecordNotFound: (2, 'XYZXYZ_ERR_RECORD_NOT_FOUND', 'src/XYZ/XYZ/XYZ.c', 113)
DEBUG 2019-04-18 12:04:11,701 decorators 17 140543334016768 Request Body:{}
DEBUG 2019-04-18 12:04:11,701 event_viewsets 17 140543334016768 Site ID not present as query parameter, doing lookup in XYZ_MAP ...
DEBUG 2019-04-18 12:04:11,706 event_viewsets 17 140543334016768 Returning response with client cookie:XYZ
WARNING 2019-04-18 10:39:26,416 base 19 140544438826752 Not Found: /XYZ/XYZ.exp
ERROR 2019-04-19 11:38:48,148 event_viewsets 16 140543468234496 Signature has expired
Traceback (most recent call last):
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 131, in XYZXYZ
user_bins = XYZXYZ_ERR_RECORD_NOT_FOUND.XYZXYZ(XYZXYZ)
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 203, in XYZXYZ
raise e
File "/opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 199, in XYZXYZ
key, meta, bin = self.XYZ.get(key)
File "/usr//opt/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX/", line 284, in verify
'signatures' + repr(self.verifylog))
xyz.xyz.XYZ: Verification failed for all XYZ['Failed: [Invalid("Verification failed {XYZ('Temporary Text.',)}",)]']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/XXXXXXXXX/XXXXXXXXX/XXXXXXXXX/views/event_viewsets.py", line 269, in track
client_cookie, is_new, _ = XYZ_or_XYZ_XYZ(request, UID_COOKIE_NAME)
File "/opt/XXXXXXXXX/XXXXXXXXX/XXXXXXXXX/utils.py", line 164, in XYZ_or_XYZ_XYZ
XYZ_or_XYZ_XYZ = ClientCookie.get_decoded_cookie(arbitrary_cookie).XYZ_or_XYZ_XYZ
File "/opt/XXXXXXXXX/XXXXXXXXX/XXXXXXXXX/utils.py", line 69, in get_decoded_cookie
payload = jwt_decode_handler(encoded_cookie)
File "/usr/XXXXXXXXX/XXXXXXXXX/XXXXXXXXX/api.xx", line 175, in _XYZ_exp
raise ExpiredSignatureError('XYZ has XYZ')
XYZ.XYZ.ExpiredSignatureError: XYZ XYZ XYZ
WARNING 2019-04-19 11:40:52,260 base 17 140543459579648 Not Found: /favicon.ico
DEBUG 2019-04-19 11:45:40,707 publisher_viewsets 17 140543435130624 Getting deal record for subdomain:XYZ.com
\\

2

Any suggestions

3

Could you please share your filebeat configuration you are testing with? Also, please use </> when sharing your logs, as simply pasting it here, does not preserve whitespaces.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.