Logtash Pattern not working


(charles) #1
input {
 beats {
         port => 5044
   }    
}
filter {
    grok {
        match => ["mesage", "ts:(?<date>(([0-9]+)-*)+ ([0-9]+-*)+ [A-Z]+)\|INFO:%{{LOGLEVEL:LEVE}\|WebContainer:%{WORD:WebContainer}\|request:%{WORD:request}\|jsessionid:%{WORD:jsessionid}"]
    }
    date {
        match => ["ts","yyyy-MM-dd HH-mm-ss-SSS"]
        target => "@timestamp"
    }

date {
        match => ["date","yyyy-MM-dd HH-mm-ss-SSS z"]
        target => "@timestamp"
    }
  }
	
  output {
    elasticsearch {
    hosts => "localhost:9200"
    #index => "%{[@metadata]}-%{+YYYY.MM.dd}"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
}
}

(Magnus Bäck) #2

Is the field that you want to parse with grok really named mesage?


(charles) #3

My log snippet
"input_type": "log",
"message": "2017-10-12 16:36:14,716 INFO [WebContainer : 2] x.x.x.x.x.OperationController [OperationController.java:113] processRequest UserInfo request: UserInfo{accessAccount='null', attributes={}, creationDate=Thu Oct 12 16:35:40 SAST 2017, userLocale=null, deviceLocalDateTime=null, deviceOSName='ANDROID', deviceOSVersion='null', appVersion='null', authenticated=false, userNumber='null', localCurrencyCode='x', timeZoneOffSet='0', phoneNumber=null, countryCode='null', activityRefNo='null', sessionId='xxxxxxxxxxx', lastLoginTime='null', deviceId='xxxxxxxxxxx', deviceModelName='null', originatorIPAddress='null', channel='S', ipAddress='xxxxxxxxxxx', localIpAddress='xxxxxxxxxxx', correlationId='null', email='null', iVal='null', jsessionid='xxxxxxxxxxx:xxxxxxxxx', enterpriseSessionId='null', wfpt='null', xfpt='null', nonce='ovLPayS4JzLaB3T6wCzIw-LpLaqJZe1uxm4UsoeVipw', userAgent='null', OpList=[]} ",Preformatted text


(Magnus Bäck) #4

So your event contains a message field but you've configured your grok filter to parse a mesage field that probably doesn't exist.


(charles) #5
i updated the Grok filter 
I am getting this response

herewith my log - please help getting a pattern

2017 - 10 - 20 08: 53: 39, 785[WebContainer: 1] ACTIVITY OUT: xxxxxx ErrorCode = 00000 Field = 00000 Exception = null JSON = {
"field1": "xxxxxx",
"field2": "00000",
"field3": "value",
"field4": "value",
"Parent": [{
"child1": "value",
"child2": "value",
"child3": "value",
"Parent": {
"child1": "value",
"child2": "value"
},
"Parent1": {
"child1": "value",
"child2": "value"
},
"field5": "value",
"field6": "value",
"field7": "value",
"field8": "value",
"field9": "value",
"field10": "value",
"Parent2": {
"child1": "Value",
"child2": "Value"
}
}, {
"child1": "value",
"child2": "value",
"child3": "value",
"Parent": {
"child1": "value",
"child2": "value"
},
"Parent1": {
"child1": "value",
"child2": "value"
},
"field5": "value",
"field6": "value",
"field7": "value",
"field8": "value",
"field9": "value",
"field10": "value",
"Parent3": {
"child1": "value",
"child2": "value"
}
}, {
"child1": "value",
"child2": "value",
"child3": "value",
"Parent": {
"child1": "value",
"child2": "value"
},
"Parent1": {
"child1": "value",
"child2": "value"
},
"field5": "value,
"field5": "value",
"field5": "value",
"field5": "value",
"field5": "value",
"field5": "value",
"Parent4": {
"child1": "value",
"child2": "value"
}
}


(Magnus Bäck) #6

match => ["mesage", "ts:(?(([0-9]+)-)+ ([0-9]+-)+ ...

No! Your configuration still says mesage instead of message. Over and out.


(charles) #7
rectified - uploaded the pattern

geytting this errror

	"tags": [
		"beats_input_codec_plain_applied",
		"_grokparsefailure",
		"_geoip_lookup_failure"

(charles) #8
 input {
 	beats {
 		port => 5044
 	}
 }
 filter {
 	grok {
 		match => ["message", "ts:(?<date>(([0-9]+)-)+ ([0-9]+-)+ [A-Z]+)|INFO:%{{LOGLEVEL:LEVE}|WebContainer:%{WORD:WebContainer}|request:%{WORD:request}|response:%{WORD:response}||jsessionid:%{WORD:jsessionid}"]
 	}
 	date {
 		match => ["ts", "yyyy-MM-dd HH-mm-ss-SSS"]
 		target => "@timestamp"
 	}

 	date {
 		match => ["date", "yyyy-MM-dd HH-mm-ss-SSS z"]
 		target => "@timestamp"
 	}
 }

 output {
 	elasticsearch {
 		hosts => "localhost:9200"
 		index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
 		document_type => "%{[@metadata][type]}"
 	}
 }

(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.