Multiline under multiline filebeat

[alex_vermex]

Hi,
I split each log using multiline filebeat:

- type: log
  enabled: true
  paths:
    - D:\elastic_stack\journal\*
  fields:
        kafka_topic: "kafka-topic-1"
  
  multiline.type: pattern
  multiline.pattern: '^\s*\*TR\sSTART\*\s*'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 1000

So with that it gives me every *TR START* gives me a document, sometimes I found a document like this:

*TR START*                
 AAAA INS                          
AAAA: 666677777                 
DATE 20-07-22    TIME 09:21:41          
 09:21:47 PI                  
 09:21:53 AT             
 09:21:55 OP            
 09:21:55 GEN               
                                        
******************                      
DATE       HEURE    A                 
20/07/22   09:21    0000001          
  AAAA:77777777777                  
MO  :       10.000             
TR. NO:     3998                     
                                        
  TR REFUSEE                   
 09:22:02 PI                   
 09:22:03 AT           
 09:22:05 OP          
 09:22:05 GEN                   
                                        
******************                      
DATE       HEURE    A                 
20/07/22   09:21    00000333          
AAAA:0000999999                  
MO  :       20            
TR. NO:     3999                     
                                        
  TR REFUSEE                   
 09:22:12 PI                  
 09:22:12 AT             
 09:22:14 OP           
 09:22:15 GEN               

******************                      
DATE       HEURE    A                 
20/07/22   09:21    5555555         
AAAA:99944422000                  
MO  :       10           
TR. NO:     4000                     
                                        
  TR REFUSEE

So I want to split it with multiline TR REFUSEE it means give me a new document, for this example so it would have to add me 3 new documents because I have 3 TR REFUSEE

Any help would be sincerely appreciate!
Thanks!

[alex_vermex]

I mean after splitting the hole file with TR START if there is TR REFUGEE in one document do the multiline with 20 lines for example.
i tried this

- type: log
  enabled: true
  paths:
    - D:\elastic_stack\journal\*
  fields:
        kafka_topic: "kafka-topic-1"
  
  multiline.type: pattern
  multiline.pattern: '^\s*\*TR\sSTART\*\s*'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 1000
  multiline.type: pattern
  multiline.pattern: '^\s*TR\sREFUSEE\s*'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 20

but it split me the whole file with TR REFUSED i don't know how it works

Any help would be sincerely appreciate!
Thanks!

[jsoriano (Jaime Soriano)]

Hi,

Would it work if you configure a pattern that matches both strings that you expect to start a document?

Something like this (not tested):

  multiline.type: pattern
  multiline.pattern: '^\s*(\*TR\sSTART\*|\s*TR REFUSEE)\s*'`
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 1000