Grok filter for IIS no working


(Carlos Eduardo Bodini Ribeiro) #1

hey there,
my grok filter for IIS logs is not working on logstash, however it worok on kibana debugger:

input {
beats {
port => 5044
type => "log"

port=> 5044
type => "iis"
}
filter {
if [type] == "iis" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} (?:-|"%{URIPATH:CS-URI-Query}") %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
remove_field => ["message"]

}

}
}

output {
elasticsearch {
hosts => "10.175.142.92:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}}

in Kibana the logs are showing like this:

{
"_index": "filebeat-2019.01.14",
"_type": "doc",
"_id": "X8LOTGgBdSYGhbv_YsLO",
"_version": 1,
"_score": null,
"_source": {
"beat": {
"hostname": "U4VMUSPUISITG11",
"name": "U4VMUSPUISITG11",
"version": "6.5.4"
},
"input": {
"type": "log"
},
"host": {
"os": {
"platform": "windows",
"family": "windows",
"build": "9200.22620",
"version": "6.2"
},
"name": "U4VMUSPUISITG11",
"id": "e2b6035c-59e1-49c1-be70-dffd00525749",
"architecture": "x86_64"
},
"@version": "1",
"offset": 464770,
"@timestamp": "2019-01-14T14:40:19.532Z",
"source": "D:\Logfiles\W3SVC1\u_ex19011414.log",
"prospector": {
"type": "log"
},
"tags": [
"beats_input_codec_plain_applied"
],
"type": "log",
"message": "2019-01-14 14:39:47 W3SVC1 U4VMUSPUISITG11 10.160.227.166 GET /Customer/Load a=4002229&p=04&b=3696&ac=&ro=EBD%20-%20Queue%20Sales%20Rep%20Brazil 80 americas\ServiceUspCPNonPrd 10.175.140.244 HTTP/1.1 Mozilla/5.0+(X11;+Linux+x86_64;+Catchpoint)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 - - u4vmuspuisitg11.olqa.preol.dell.com 302 0 0 716 527 946"
},
"fields": {
"@timestamp": [
"2019-01-14T14:40:19.532Z"
]
},
"sort": [
1547476819532
]
}

i would appreciate some help here!


#2

You are trying to match (?:-|"%{URIPATH:CS-URI-Query}") to

"CS-URI-Query" => "a=4002229&p=04&b=3696&ac=&ro=EBD%20-%20Queue%20Sales%20Rep%20Brazil"

That is not a URIPATH. It is not even a URIPARAM (since that needs a leading ?}. Try replacing that whole term with %{NOTSPACE}.

Also, for some items you may want to convert to int. For example

%{NUMBER:SC-Bytes:int} %{NUMBER:CS-Bytes:int}

Personally I do not convert fields like S-Port or SC-Status to int, since I will never need to do calculations on them (would you ever care about the average port number? :slight_smile: )


(Carlos Eduardo Bodini Ribeiro) #3

hey Badger,
thanks for your reply. The issue here is not the filter itself, but why is not being applied in the logs
check this screenshots:
from how the log is showing up in kibana

the filter actually works, in grok debugger, inside Kibana:


#4

The pattern you have will work if there is no query in the URL. It will not work if there are URI parameters.

It is not getting applied because the event has type "log", not type "iis".


(Carlos Eduardo Bodini Ribeiro) #5

i see... and Could i change it to 'iis'? because in my filebeat.yml, i have the "document_type: iis" line and in logstash the type=> "iis"

image
image


#6

Is that really what your config looks like?


(Carlos Eduardo Bodini Ribeiro) #7

right now is looking like this:
image

but still the same scenario


#8

In Kibana that is the type you had before. What does it show now?


(Carlos Eduardo Bodini Ribeiro) #9

still the same thing,
{
"_index": "filebeat-2019.01.15",
"_type": "doc",
"_id": "zc4_UmgBdSYGhbv_GS89",
"_version": 1,
"_score": null,
"_source": {
"input": {
"type": "log"
},
"beat": {
"name": "xxxxxxx",
"version": "6.5.4",
"hostname": "xxxxxxxxxxx"
},
"host": {
"architecture": "x86_64",
"os": {
"family": "windows",
"platform": "windows",
"build": "9200.22620",
"version": "6.2"
},
"name": "xxxxxxxxx",
"id": "65fd3db9-0bfe-45b5-bd81-bdc27a33af3f"
},
"type": "log",
"@version": "1",
"prospector": {
"type": "log"
},
"message": "2019-01-15 16:01:00 W3SVC1 xxxxxxxxxx10.160.227.164 GET /Content/Themes/base/images/JQGrid/ui-icons_469bdd_256x240.png - 80 americas\xxxxxxxxxx 10.175.140.246 HTTP/1.1 Mozilla/5.0+(X11;+Linux+x86_64;+Catchpoint)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 HasAppSupportRole=true http://xxx.x/Content/Themes/base/JQGrid/jquery.jqgrid-ui.css xxxxxx.x 200 0 0 4639 524 31",
"tags": [
"beats_input_codec_plain_applied"
],
"source": "D:\logfiles\W3SVC1\u_ex19011516.log",
"@timestamp": "2019-01-15T16:01:32.435Z",
"offset": 205434
},
"fields": {
"@timestamp": [
"2019-01-15T16:01:32.435Z"
]
},
"sort": [
1547568092435
]
}


#10

type on a beat input is ignored if the incoming event already has a type field.

In filebeat, document_type, which sets _type, was removed in 6.0. Use a custom field as shown here.


(Carlos Eduardo Bodini Ribeiro) #11

yep, now the type is showing as IIS
image

however the "message" filed still not being grok by my filter, maybe is something wrong with my beats.conf?

input {
  beats {
   port => 5044
   type => "iis"
   }}
  
filter {
  if [type] == "iis" {
   grok {
     match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} (?:-|\"%{URIPATH:CS-URI-Query}\") %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
	 }
  }
}
 


output {
  elasticsearch {
    hosts => "10.175.142.92:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

#12

Are you getting a _grokparsefailure tag?


(Carlos Eduardo Bodini Ribeiro) #13

actually not, no error messages


#14

Enable debug logging. Do you see a line like

[DEBUG][logstash.filters.grok    ] Running grok filter {:event=>#<LogStash::Event:0x4658baf8>}

If not, then your [type] == "iis" still is not matching. But I cannot think of anything else to explain why.


(Carlos Eduardo Bodini Ribeiro) #15

there's no [DEBUG][logstash.filters.grok ] errors in the log file, it seams the filter is not being triggered by some reason