Question in Multi line grok pattern

Elastic Stack Logstash
1

hi there

this is my #beats:filebeat config file that read and send logs to output #logstash:

filebeat.prospectors:
    - input_type: log

      paths:
        - D:\log\my-log01.log

      tags: ["log01"]

      fields:
        log-01: true

    #================================
    - input_type: log

      paths:
        - D:\log\my-log02.log

      tags: ["log02"] #***tags***

      fields:
        log-02: true #***field****

and... some more files like this....

here is my #logstash config file that sets (path.config: ..\config\my-pipeline.conf) to read from my-pipeline:

my-pipeline:

input {
  beats {
		type => "log"
		port => 5044
    }
}

filter {
	if [log-01] == "true" { #***conditional statement***
		grok {
			patterns_dir => "./patterns"
			break_on_match => false
			match =>  [
						"message" , "%{LOGINFO} -> %{NUMBERMSGID}->*%{DATA:action}->MpcRequest{id=%{DATA:ReqID}, appId=%{INT:appID}, appInfo=%{QS:appInfo}, opCode=%{DATA:OpCode}, sessionId=%{DATA:SessionID}, mobileNo='%{INT:MobileNo}', messageBody=%{QS:MessageBody}, opCode=%{DATA:OpCode}, ctm=%{DATA:CTM}, udh=%{DATA:UDH}, keyword=%{DATA:keyword}, messageId=%{DATA:MessageID}, host.IP=Host{id=%{DATA:ID}, ip='%{IP:ip}'}, httpHeader.RemoteAddr=Header{id=%{DATA:HeaderID}, remoteAddr='%{IP:RemoteIP}:%{INT:Port}', xForwardedFor='%{DATA:xForwardedFor}'}, smsProviderSmsProvider{id=%{DATA:SMSProviderID}, userName=%{DATA:UserName}, smsNumber=%{DATA:SMSNumber}}}, messageBody=%{GREEDYDATA:MessageBody}}",
						"message" , "%{LOGINFO} -> %{NUMBERMSGID}->=%{WORD:health}=%{DATA:action}==>RSResponse{tranId= %{DATA:TeranID}, status=(?<Status>\w+\(\d+\)), opCode=(?<OP-Code>\w+\(\d+\)), securityStatus=%{NUMBER:securityStatus}, description=%{QUOTEDSTRING:description}, serverTime=%{QS}, advertise=%{DATA:advertise}, extraData=%{GREEDYDATA:ExteraData}}",
						"message" , "%{LOGINFO} -> %{NUMBERMSGID}->%{DATA:action}->(?<Length>\(\w+:\d+\)): %{GREEDYDATA:msg}",
						"message" , "%{LOGINFO} -> %{NUMBERMSGID}->%{GREEDYDATA:msg}",
						"message" , "%{LOGINFO} -> %{GREEDYDATA:msg}"
					  ]
			tag_on_failure => ["log01-Failure"]
		}
	}
	if [log-02] == "true" {
		grok {
			patterns_dir => "./patterns"
			break_on_match => false
			match =>  [
						"message" , "%{LOGINFO} -> %{NUMBERMSGID}->*%{DATA:action}->MpcRequest{id=%{DATA:ReqID}, appId=%{INT:appID}, appInfo=%{QS:appInfo}, opCode=%{DATA:OpCode}, sessionId=%{DATA:SessionID}, mobileNo='%{INT:MobileNo}', messageBody=%{QS:MessageBody}, opCode=%{DATA:OpCode}, ctm=%{DATA:CTM}, udh=%{DATA:UDH}, keyword=%{DATA:keyword}, messageId=%{DATA:MessageID}, host.IP=Host{id=%{DATA:ID}, ip='%{IP:ip}'}, httpHeader.RemoteAddr=Header{id=%{DATA:HeaderID}, remoteAddr='%{IP:RemoteIP}:%{INT:Port}', xForwardedFor='%{DATA:xForwardedFor}'}, smsProviderSmsProvider{id=%{DATA:SMSProviderID}, userName=%{DATA:UserName}, smsNumber=%{DATA:SMSNumber}}}, messageBody=%{GREEDYDATA:MessageBody}}",
						"message" , "%{LOGINFO} -> %{NUMBERMSGID}->=%{WORD:health}=%{DATA:action}==>RSResponse{tranId= %{DATA:TeranID}, status=(?<Status>\w+\(\d+\)), opCode=(?<OP-Code>\w+\(\d+\)), securityStatus=%{NUMBER:securityStatus}, description=%{QUOTEDSTRING:description}, serverTime=%{QS}, advertise=%{DATA:advertise}, extraData=%{GREEDYDATA:ExteraData}}",
						"message" , "%{LOGINFO} -> %{NUMBERMSGID}->%{DATA:action}->(?<Length>\(\w+:\d+\)): %{GREEDYDATA:msg}",
						"message" , "%{LOGINFO} -> %{NUMBERMSGID}->%{GREEDYDATA:msg}",
						"message" , "%{LOGINFO} -> %{GREEDYDATA:msg}"
					  ]
			tag_on_failure => ["log02-Failure"]
		}
	}
}

output {
	stdout{
		codec => rubydebug
	}
elasticsearch {
    hosts => ["localhost:9200"]
}     
}

and there is also some custom pattern that i makes them using RegEX in ./pattern folder.:sunglasses:

  • there is no problem in stashing the logs without conditional statement.

Questions :blush:

  1. did i correctly use of different multiline grok patterns for different log files ?
  2. how can i check the tag names (mention by #***tags*** in code) or field names (mention by #***field*** in code) that i defined in #beats:filebeat config file at conditional statement (mentiond by#***conditional statement***) ?
2

@magnusbaeck need your help bro

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.