Filestream parsers multiline pattern parse all file in one hit althoutgh I test my pattern in The Go Playground

Here is the parser in filestream input

parsers:
    - multiline:
        type: pattern
        pattern: '^DEBUG|INFO|ERROR'
        negate: true
        match: after
        # flush_pattern: '^\s*$'

and this is the log file

INFO  (null) 2022-12-04 10:49:38,986 - Loading BBSF Module...............

------------------------
DEBUG (null) 2022-12-04 10:49:44,229 - Database Migration Initialized

------------------------
DEBUG (null) 2022-12-04 10:49:44,230 - Reading configMigration: ID: 0 - Client:InitialCreate - BBSF:

------------------------
DEBUG (null) 2022-12-04 10:49:46,395 - Try Load ProviderID:1 - Name:SqlProvider - Type:Bnsights.Providers.File.Sql.SqlProvider, Bnsights.Providers.File.Sql

------------------------
DEBUG (null) 2022-12-04 10:49:46,520 - Success loading - Type:Bnsights.Providers.File.Sql.SqlProvider, Bnsights.Providers.File.Sql

------------------------
DEBUG (null) 2022-12-04 10:49:46,521 - Try Load ProviderID:2 - Name:ExchangeProvider - Type:Bnsights.Providers.Email.Smtp.SmtpProvider, Bnsights.Providers.Email.Smtp

------------------------
DEBUG (null) 2022-12-04 10:49:46,543 - Success loading - Type:Bnsights.Providers.Email.Smtp.SmtpProvider, Bnsights.Providers.Email.Smtp

------------------------
DEBUG (null) 2022-12-04 10:49:46,543 - Try Load ProviderID:3 - Name:EWSProvider - Type:Bnsights.Providers.Email.EWS.EwsProvider, Bnsights.Providers.Email.EWS

------------------------
DEBUG (null) 2022-12-04 10:49:46,562 - Success loading - Type:Bnsights.Providers.Email.EWS.EwsProvider, Bnsights.Providers.Email.EWS

------------------------
DEBUG (null) 2022-12-04 10:49:46,562 - Try Load ProviderID:4 - Name:NoOpProvider - Type:Bnsights.Providers.SMS.NoOp.NoOpProvider, Bnsights.Providers.SMS.NoOp

------------------------
DEBUG (null) 2022-12-04 10:49:46,573 - Success loading - Type:Bnsights.Providers.SMS.NoOp.NoOpProvider, Bnsights.Providers.SMS.NoOp

------------------------
DEBUG (null) 2022-12-04 10:49:46,574 - Try Load ProviderID:5 - Name:ServiceRequestsSqlProvider - Type:Bnsights.Providers.File.Sql.SqlProvider, Bnsights.Providers.File.Sql

------------------------
DEBUG (null) 2022-12-04 10:49:46,602 - Success loading - Type:Bnsights.Providers.File.Sql.SqlProvider, Bnsights.Providers.File.Sql

------------------------
ERROR (null) 2022-12-04 10:49:59,329 - Error validating token
Microsoft.IdentityModel.Tokens.SecurityTokenExpiredException: IDX10223: Lifetime validation failed. The token is expired. ValidTo: 'System.DateTime', Current time: 'System.DateTime'.
   at Microsoft.IdentityModel.Tokens.Validators.ValidateLifetime(Nullable`1 notBefore, Nullable`1 expires, SecurityToken securityToken, TokenValidationParameters validationParameters)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateLifetime(Nullable`1 notBefore, Nullable`1 expires, JwtSecurityToken jwtToken, TokenValidationParameters validationParameters)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateTokenPayload(JwtSecurityToken jwtToken, TokenValidationParameters validationParameters)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateToken(String token, TokenValidationParameters validationParameters, SecurityToken& validatedToken)
   at ClientApp.BLL.ApiAuth.JwtAuthManager.GetPrincipal(String token) in D:\GIT\UCP\ClientApp.BLL\ApiAuth\JwtAuthManager.cs:line 78

------------------------
INFO  (null) 2022-12-04 10:49:59,364 - JWT Token Error for token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1bmlxdWVfbmFtZSI6IlRSQUFkbWluIiwibmJmIjoxNjcwMDQ2MDAyLCJleHAiOjE2NzAxMzI0MDIsImlhdCI6MTY3MDA0NjAwMn0.Nf07-syA4Y31ezbc3uOj-RR9bn5FJVbP5n9P59_anx8 on request https://localhost:44399/Auth/Login

------------------------
DEBUG 05FF9A08-FEF1-4B54-A82B-B53C117BA032 2022-12-04 10:49:59,608 - JWT Token generated for username: TRAAdmin - Expiry: 05/12/2022 08:49:59

------------------------
INFO  (null) 2022-12-04 10:50:28,025 - Daily Job starting..

------------------------
INFO  (null) 2022-12-04 10:50:28,117 - Daily Job stopping..

------------------------
INFO  (null) 2022-12-04 12:00:29,441 - Daily Job starting..

------------------------
INFO  (null) 2022-12-04 12:00:29,492 - Daily Job stopping..

------------------------

For the pattern, don't you need a group?

'^(DEBUG|INFO|ERROR)'

Either ( or [.

1 Like

You cannot use exclude_lines, there is skip_newline and if you want to remove -----, or also debug, not accept like this: ^(?!DEBUG|INFO|ERROR|!-+).*$,
The processor is optional for removing -----

- type: filestream
  id: xlog
  enabled: true
  processors:
  - replace:
      fields:
        - field: "message"
          pattern: "------------------------"
          replacement: ""
      ignore_missing: true
      fail_on_error: false
  parsers:
    - multiline:
        type: pattern
        pattern: '^(DEBUG|INFO|ERROR)'        
        negate: true
        match: after
        skip_newline: true
  paths:
    - /path/file.log

Sorry I delete my last response by mistake
That was my response
Thanks for your answer, Ok it works without a group, I found the problem with me, It was I should rerun the setup command, I changed my pattern and looked for differences and nothing happened but after running the setup command it worked with me. One more question please, Can I exclude both empty lines and ------------------------ from my log by using pattern?

Thanks for your answers

Yes both will be excluded, pls test.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.