Grok Parse Failure

Elastic Stack Logstash
1

Hello Logstash Community!

I'm having trouble parsing my logs in Kibana. The logs are being sent to log stash (file beat) just fine. Additionally, the grok patterns/filters are showing up in Kibana (settings--Indicies). In the discover tab, the logs are not being filtered through the fields i've specified in the grok pattern. I'm getting a _grokparsefailure in the tags filter.

Please let me know what you need from me to help solve this issue!

Thank you in advance!

2

@Sully1 Can you post your logstash configuration along with some of input data which you are trying to parse through logstash.

3

Sure. Conf & Input data below.

#1_beats_inputs.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

#2_iis_filters.conf
filter {
if [message] =~ "^#" {drop {}}

    #IIS
	if [type] == "IISLog" {
            grok {
                  match => [
		"message","(%{TIMESTAMP_ISO8601:log_timestamp})\s(?<Server_Sitename>(.*?))\s(?<Server_IP>(.*?))\s(?<HTTP_Request_Method>(.*?))\s(?<IIS_Client_Query>(.*?))\s(?<Client_Stem>(.*?))\s(?<Server_Port>(.*?))\s(?<Client_ID>(.*?))\s(?<Client_IP>(.*?))\s(?<User_Agent>(.*?))\s(?<Referer>(.*?))\s(?<HTTP_Status_Code>(?:[4-5_.-]+[0-9_.-]+[0-9_.-]))\s(?<Server_Substatus>(.*?))\s(?<Server_Win32_Status>(.*?))\s(%{GREEDYDATA:Request_Time_Elapsed})",
		"message","(%{TIMESTAMP_ISO8601:log_timestamp})\s(?<Server_IP>(.*?))\s(?<HTTP_Request_Method>(.*?))\s(?<IIS_Client_Query>(.*?))\s(?<Client_Stem>(.*?))\s(?<Server_Port>(.*?))\s(?<Client_ID>(.*?))\s(?<Client_IP>(.*?))\s(?<User_Agent>(.*?))\s(?<Referer>(.*?))\s(?<HTTP_Status_Code>(?:[4-5_.-]+[0-9_.-]+[0-9_.-]))\s(?<Server_Substatus>(.*?))\s(?<Server_Win32_Status>(.*?))\s(%{GREEDYDATA:Request_Time_Elapsed})",
		"message","(%{TIMESTAMP_ISO8601:log_timestamp})\s(?<Server_IP>(.*?))\s(?<HTTP_Request_Method>(.*?))\s(?<IIS_Client_Query>(.*?))\s(?<Client_Stem>(.*?))\s(?<Server_Port>(.*?))\s(?<Client_ID>(.*?))\s(?<Client_IP>(.*?))\s(?<User_Agent>(.*?))\s(?<HTTP_Status_Code>(?:[4-5_.-]+[0-9_.-]+[0-9_.-]))\s(?<Server_Substatus>(.*?))\s(?<Server_Win32_Status>(.*?))\s(%{GREEDYDATA:Request_Time_Elapsed})"
	         	]

                  add_tag => "IIS"}
			date {
            match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
            timezone => "Etc/UCT"
		}
    }

#3_output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}

#input data

Software: Microsoft Internet Information Services 8.5
Version: 1.0
Date: 2016-07-29 12:11:35
Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2016-07-29 12:11:35 xx.xxx.xx.xxx GET /UI - 5555 - xx.xxx.xx.xxx Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:45.0)+Gecko/20100101+Firefox/45.0 - 401 2 5 225
2016-07-29 12:11:35 xx.xxx.xx.xxx GET /UI - 5555 xxxx\xxxx xx.xxx.xx.xxx Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:45.0)+Gecko/20100101+Firefox/45.0 - 301 0 0 14
2016-07-29 12:11:35 xx.xxx.xx.xxx GET /UI/ - 5555 xxxx\xxxx xx.xxx.xx.xxx Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:45.0)+Gecko/20100101+Firefox/45.0 - 200 0 0 2
2016-07-29 12:11:35 xx.xxx.xx.xxx GET /UI/js/app.js - 5555 - xx.xxx.xx.xxx Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:45.0)+Gecko/20100101+Firefox/45.0 http://xxx.rs.xxx.org:5555/UI/ 401 2 5 0