Filestream parsers multiline pattern parse all file in one hit althoutgh I test my pattern in The Go Playground

Elastic Stack Beats
1

Here is the parser in filestream input

parsers:
    - multiline:
        type: pattern
        pattern: '^DEBUG|INFO|ERROR'
        negate: true
        match: after
        # flush_pattern: '^\s*$'

and this is the log file

INFO  (null) 2022-12-04 10:49:38,986 - Loading BBSF Module...............

------------------------
DEBUG (null) 2022-12-04 10:49:44,229 - Database Migration Initialized

------------------------
DEBUG (null) 2022-12-04 10:49:44,230 - Reading configMigration: ID: 0 - Client:InitialCreate - BBSF:

------------------------
DEBUG (null) 2022-12-04 10:49:46,395 - Try Load ProviderID:1 - Name:SqlProvider - Type:Bnsights.Providers.File.Sql.SqlProvider, Bnsights.Providers.File.Sql

------------------------
DEBUG (null) 2022-12-04 10:49:46,520 - Success loading - Type:Bnsights.Providers.File.Sql.SqlProvider, Bnsights.Providers.File.Sql

------------------------
DEBUG (null) 2022-12-04 10:49:46,521 - Try Load ProviderID:2 - Name:ExchangeProvider - Type:Bnsights.Providers.Email.Smtp.SmtpProvider, Bnsights.Providers.Email.Smtp

------------------------
DEBUG (null) 2022-12-04 10:49:46,543 - Success loading - Type:Bnsights.Providers.Email.Smtp.SmtpProvider, Bnsights.Providers.Email.Smtp

------------------------
DEBUG (null) 2022-12-04 10:49:46,543 - Try Load ProviderID:3 - Name:EWSProvider - Type:Bnsights.Providers.Email.EWS.EwsProvider, Bnsights.Providers.Email.EWS

------------------------
DEBUG (null) 2022-12-04 10:49:46,562 - Success loading - Type:Bnsights.Providers.Email.EWS.EwsProvider, Bnsights.Providers.Email.EWS

------------------------
DEBUG (null) 2022-12-04 10:49:46,562 - Try Load ProviderID:4 - Name:NoOpProvider - Type:Bnsights.Providers.SMS.NoOp.NoOpProvider, Bnsights.Providers.SMS.NoOp

------------------------
DEBUG (null) 2022-12-04 10:49:46,573 - Success loading - Type:Bnsights.Providers.SMS.NoOp.NoOpProvider, Bnsights.Providers.SMS.NoOp

------------------------
DEBUG (null) 2022-12-04 10:49:46,574 - Try Load ProviderID:5 - Name:ServiceRequestsSqlProvider - Type:Bnsights.Providers.File.Sql.SqlProvider, Bnsights.Providers.File.Sql

------------------------
DEBUG (null) 2022-12-04 10:49:46,602 - Success loading - Type:Bnsights.Providers.File.Sql.SqlProvider, Bnsights.Providers.File.Sql

------------------------
ERROR (null) 2022-12-04 10:49:59,329 - Error validating token
Microsoft.IdentityModel.Tokens.SecurityTokenExpiredException: IDX10223: Lifetime validation failed. The token is expired. ValidTo: 'System.DateTime', Current time: 'System.DateTime'.
   at Microsoft.IdentityModel.Tokens.Validators.ValidateLifetime(Nullable`1 notBefore, Nullable`1 expires, SecurityToken securityToken, TokenValidationParameters validationParameters)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateLifetime(Nullable`1 notBefore, Nullable`1 expires, JwtSecurityToken jwtToken, TokenValidationParameters validationParameters)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateTokenPayload(JwtSecurityToken jwtToken, TokenValidationParameters validationParameters)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateToken(String token, TokenValidationParameters validationParameters, SecurityToken& validatedToken)
   at ClientApp.BLL.ApiAuth.JwtAuthManager.GetPrincipal(String token) in D:\GIT\UCP\ClientApp.BLL\ApiAuth\JwtAuthManager.cs:line 78

------------------------
INFO  (null) 2022-12-04 10:49:59,364 - JWT Token Error for token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1bmlxdWVfbmFtZSI6IlRSQUFkbWluIiwibmJmIjoxNjcwMDQ2MDAyLCJleHAiOjE2NzAxMzI0MDIsImlhdCI6MTY3MDA0NjAwMn0.Nf07-syA4Y31ezbc3uOj-RR9bn5FJVbP5n9P59_anx8 on request https://localhost:44399/Auth/Login

------------------------
DEBUG 05FF9A08-FEF1-4B54-A82B-B53C117BA032 2022-12-04 10:49:59,608 - JWT Token generated for username: TRAAdmin - Expiry: 05/12/2022 08:49:59

------------------------
INFO  (null) 2022-12-04 10:50:28,025 - Daily Job starting..

------------------------
INFO  (null) 2022-12-04 10:50:28,117 - Daily Job stopping..

------------------------
INFO  (null) 2022-12-04 12:00:29,441 - Daily Job starting..

------------------------
INFO  (null) 2022-12-04 12:00:29,492 - Daily Job stopping..

------------------------
2

For the pattern, don't you need a group?

'^(DEBUG|INFO|ERROR)'

Either ( or [.

1 Like
4

You cannot use exclude_lines, there is skip_newline and if you want to remove -----, or also debug, not accept like this: ^(?!DEBUG|INFO|ERROR|!-+).*$,
The processor is optional for removing -----

- type: filestream
  id: xlog
  enabled: true
  processors:
  - replace:
      fields:
        - field: "message"
          pattern: "------------------------"
          replacement: ""
      ignore_missing: true
      fail_on_error: false
  parsers:
    - multiline:
        type: pattern
        pattern: '^(DEBUG|INFO|ERROR)'        
        negate: true
        match: after
        skip_newline: true
  paths:
    - /path/file.log
5

Sorry I delete my last response by mistake
That was my response
Thanks for your answer, Ok it works without a group, I found the problem with me, It was I should rerun the setup command, I changed my pattern and looked for differences and nothing happened but after running the setup command it worked with me. One more question please, Can I exclude both empty lines and ------------------------ from my log by using pattern?

6

Thanks for your answers

7

Yes both will be excluded, pls test.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.