Devise logs

alex_vermex · August 10, 2022, 2:48pm

Hi,
i have log file it looks like this :

     *TR START*                
 AAAA INS                          
AAAA: 121212                  
DATE 20-07-22    TIME 09:18:00          
 09:18:04 PI                   
 09:18:13 AT              
 09:18:15 OP             
 09:18:16 GE                
 09:18:18 GE                  
 09:18:27 NO                 
 09:18:29 AAAA TAK                    
 09:18:32 NOTES PR 0,1,0,0       
 09:18:32 NOTES TAKEN                   

******************                      
DATE       HEURE    A                 
20/07/22   09:17    333333         
AAAA:121212XX1212                  
MO  :       30            
TR. NO:     3997                     
 09:18:42 TR END                             
     *TR START*                
 AAAA INS                          
AAAA: 666677777                 
DATE 20-07-22    TIME 09:21:41          
 09:21:47 PI                  
 09:21:53 AT             
 09:21:55 OP            
 09:21:55 GEN               
                                        
******************                      
DATE       HEURE    A                 
20/07/22   09:21    0000001          
  AAAA:77777777777                  
MO  :       10.000             
TR. NO:     3998                     
                                        
  TR REFUSEE                   
 09:22:02 PI                   
 09:22:03 AT           
 09:22:05 OP          
 09:22:05 GEN                   
                                        
******************                      
DATE       HEURE    A                 
20/07/22   09:21    00000333          
AAAA:0000999999                  
MO  :       20            
TR. NO:     3999                     
                                        
  TR REFUSEE                   
 09:22:12 PI                  
 09:22:12 AT             
 09:22:14 OP           
 09:22:15 GEN               

******************                      
DATE       HEURE    A                 
20/07/22   09:21    5555555         
AAAA:99944422000                  
MO  :       10           
TR. NO:     4000                     
                                        
  TR REFUSEE                   
     *TRANSACTION START*                
 AAAA INSERTED                          
AAAA: 00333666667777                  
DATE 20-07-22    TIME 09:25:45          
 09:25:50 PI                
 09:26:07 AT              
 09:26:10 OP          
 09:26:10 GEN               
 09:26:13 GEN              
 09:26:26 NOTES STA                
 09:26:31 NOTES PRE 0,20,0,0      
                                        
******************                      
DATE       HEURE    A               
20/07/22   09:25    00000001111          
AAAA:777770000000000                  
MO  :       300           
TR. NO:     4007                     
 09:26:42 TR END

so i devise my file with filebeat multiline like this :

- type: log
  enabled: true
  paths:
    - D:\elastic_stack\journal\*
  fields:
        kafka_topic: "kafka-topic-1"
  
  multiline.type: pattern
  multiline.pattern: '^\s*\*TR\sSTART\*\s*'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 1000

so after this i have a document in elasticsearch like this :

     *TR START*                
 AAAA INS                          
AAAA: 666677777                 
DATE 20-07-22    TIME 09:21:41          
 09:21:47 PI                  
 09:21:53 AT             
 09:21:55 OP            
 09:21:55 GEN               
                                        
******************                      
DATE       HEURE    A                 
20/07/22   09:21    0000001          
  AAAA:77777777777                  
MO  :       10.000             
TR. NO:     3998                     
                                        
  TR REFUSEE                   
 09:22:02 PI                   
 09:22:03 AT           
 09:22:05 OP          
 09:22:05 GEN                   
                                        
******************                      
DATE       HEURE    A                 
20/07/22   09:21    00000333          
AAAA:0000999999                  
MO  :       20            
TR. NO:     3999                     
                                        
  TR REFUSEE                   
 09:22:12 PI                  
 09:22:12 AT             
 09:22:14 OP           
 09:22:15 GEN               

******************                      
DATE       HEURE    A                 
20/07/22   09:21    5555555         
AAAA:99944422000                  
MO  :       10           
TR. NO:     4000                     
                                        
  TR REFUSEE

I would like to know if I can also design this document with the keyword TR REFUSED I mean I want it in another document if it is possible I hope you understand me

Any help would be sincerely appreciate!
Thanks!

Badger · August 10, 2022, 3:22pm

This is not a logstash question, it is a question about alternation in a regexp in a filebeat multiline pattern. I suggest you move the question to the filebeat forum.

alex_vermex · August 10, 2022, 6:15pm

Done and thanks !

alex_vermex · August 10, 2022, 7:16pm

there is a solution for that with filebeat right? because it seems possible, I will try to explain more,
I split each log using multiline filebeat:

- type: log
  enabled: true
  paths:
    - D:\elastic_stack\journal\*
  fields:
        kafka_topic: "kafka-topic-1"
  
  multiline.type: pattern
  multiline.pattern: '^\s*\*TR\sSTART\*\s*'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 1000

So with that it gives me every *TR START* gives me a document, sometimes I found a document like this:

*TR START*                
 AAAA INS                          
AAAA: 666677777                 
DATE 20-07-22    TIME 09:21:41          
 09:21:47 PI                  
 09:21:53 AT             
 09:21:55 OP            
 09:21:55 GEN               
                                        
******************                      
DATE       HEURE    A                 
20/07/22   09:21    0000001          
  AAAA:77777777777                  
MO  :       10.000             
TR. NO:     3998                     
                                        
  TR REFUSEE                   
 09:22:02 PI                   
 09:22:03 AT           
 09:22:05 OP          
 09:22:05 GEN                   
                                        
******************                      
DATE       HEURE    A                 
20/07/22   09:21    00000333          
AAAA:0000999999                  
MO  :       20            
TR. NO:     3999                     
                                        
  TR REFUSEE                   
 09:22:12 PI                  
 09:22:12 AT             
 09:22:14 OP           
 09:22:15 GEN               

******************                      
DATE       HEURE    A                 
20/07/22   09:21    5555555         
AAAA:99944422000                  
MO  :       10           
TR. NO:     4000                     
                                        
  TR REFUSEE

So I want to split it with multiline TR REFUSEE it means give me a new document, for this example so it would have to add me 3 new documents because I have 3 TR REFUSEE

Any help would be sincerely appreciate!
Thanks!

system · September 7, 2022, 7:17pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.