Grok filter for IIS no working

Carlos_Bodini · January 14, 2019, 2:41pm

hey there,
my grok filter for IIS logs is not working on logstash, however it worok on kibana debugger:

input {
beats {
port => 5044
type => "log"

port=> 5044
type => "iis"
}
filter {
if [type] == "iis" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} (?:-|"%{URIPATH:CS-URI-Query}") %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
remove_field => ["message"]

}
}

output {
elasticsearch {
hosts => "10.175.142.92:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}}

in Kibana the logs are showing like this:

{
"_index": "filebeat-2019.01.14",
"_type": "doc",
"_id": "X8LOTGgBdSYGhbv_YsLO",
"_version": 1,
"_score": null,
"_source": {
"beat": {
"hostname": "U4VMUSPUISITG11",
"name": "U4VMUSPUISITG11",
"version": "6.5.4"
},
"input": {
"type": "log"
},
"host": {
"os": {
"platform": "windows",
"family": "windows",
"build": "9200.22620",
"version": "6.2"
},
"name": "U4VMUSPUISITG11",
"id": "e2b6035c-59e1-49c1-be70-dffd00525749",
"architecture": "x86_64"
},
"@version": "1",
"offset": 464770,
"@timestamp": "2019-01-14T14:40:19.532Z",
"source": "D:\Logfiles\W3SVC1\u_ex19011414.log",
"prospector": {
"type": "log"
},
"tags": [
"beats_input_codec_plain_applied"
],
"type": "log",
"message": "2019-01-14 14:39:47 W3SVC1 U4VMUSPUISITG11 10.160.227.166 GET /Customer/Load a=4002229&p=04&b=3696&ac=&ro=EBD%20-%20Queue%20Sales%20Rep%20Brazil 80 americas\ServiceUspCPNonPrd 10.175.140.244 HTTP/1.1 Mozilla/5.0+(X11;+Linux+x86_64;+Catchpoint)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 - - u4vmuspuisitg11.olqa.preol.dell.com 302 0 0 716 527 946"
},
"fields": {
"@timestamp": [
"2019-01-14T14:40:19.532Z"
]
},
"sort": [
1547476819532
]
}

i would appreciate some help here!

Badger · January 14, 2019, 4:33pm

You are trying to match (?:-|"%{URIPATH:CS-URI-Query}") to

"CS-URI-Query" => "a=4002229&p=04&b=3696&ac=&ro=EBD%20-%20Queue%20Sales%20Rep%20Brazil"

That is not a URIPATH. It is not even a URIPARAM (since that needs a leading ?}. Try replacing that whole term with %{NOTSPACE}.

Also, for some items you may want to convert to int. For example

%{NUMBER:SC-Bytes:int} %{NUMBER:CS-Bytes:int}

Personally I do not convert fields like S-Port or SC-Status to int, since I will never need to do calculations on them (would you ever care about the average port number? )

Carlos_Bodini · January 14, 2019, 5:09pm

hey Badger,
thanks for your reply. The issue here is not the filter itself, but why is not being applied in the logs
check this screenshots:
from how the log is showing up in kibana

the filter actually works, in grok debugger, inside Kibana:

Badger · January 14, 2019, 6:24pm

The pattern you have will work if there is no query in the URL. It will not work if there are URI parameters.

It is not getting applied because the event has type "log", not type "iis".

Carlos_Bodini · January 14, 2019, 7:25pm

i see... and Could i change it to 'iis'? because in my filebeat.yml, i have the "document_type: iis" line and in logstash the type=> "iis"

Badger · January 14, 2019, 8:08pm

Is that really what your config looks like?

Carlos_Bodini · January 15, 2019, 11:28am

right now is looking like this:

but still the same scenario

Badger · January 15, 2019, 3:15pm

In Kibana that is the type you had before. What does it show now?

Carlos_Bodini · January 15, 2019, 4:03pm

still the same thing,
{
"_index": "filebeat-2019.01.15",
"_type": "doc",
"_id": "zc4_UmgBdSYGhbv_GS89",
"_version": 1,
"_score": null,
"_source": {
"input": {
"type": "log"
},
"beat": {
"name": "xxxxxxx",
"version": "6.5.4",
"hostname": "xxxxxxxxxxx"
},
"host": {
"architecture": "x86_64",
"os": {
"family": "windows",
"platform": "windows",
"build": "9200.22620",
"version": "6.2"
},
"name": "xxxxxxxxx",
"id": "65fd3db9-0bfe-45b5-bd81-bdc27a33af3f"
},
"type": "log",
"@version": "1",
"prospector": {
"type": "log"
},
"message": "2019-01-15 16:01:00 W3SVC1 xxxxxxxxxx10.160.227.164 GET /Content/Themes/base/images/JQGrid/ui-icons_469bdd_256x240.png - 80 americas\xxxxxxxxxx 10.175.140.246 HTTP/1.1 Mozilla/5.0+(X11;+Linux+x86_64;+Catchpoint)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 HasAppSupportRole=true http://xxx.x/Content/Themes/base/JQGrid/jquery.jqgrid-ui.css xxxxxx.x 200 0 0 4639 524 31",
"tags": [
"beats_input_codec_plain_applied"
],
"source": "D:\logfiles\W3SVC1\u_ex19011516.log",
"@timestamp": "2019-01-15T16:01:32.435Z",
"offset": 205434
},
"fields": {
"@timestamp": [
"2019-01-15T16:01:32.435Z"
]
},
"sort": [
1547568092435
]
}

Badger · January 15, 2019, 4:41pm

type on a beat input is ignored if the incoming event already has a type field.

In filebeat, document_type, which sets _type, was removed in 6.0. Use a custom field as shown here.

Carlos_Bodini · January 15, 2019, 5:19pm

yep, now the type is showing as IIS

however the "message" filed still not being grok by my filter, maybe is something wrong with my beats.conf?

input {
  beats {
   port => 5044
   type => "iis"
   }}
  
filter {
  if [type] == "iis" {
   grok {
     match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} (?:-|\"%{URIPATH:CS-URI-Query}\") %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
	 }
  }
}
 


output {
  elasticsearch {
    hosts => "10.175.142.92:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Badger · January 15, 2019, 5:21pm

Are you getting a _grokparsefailure tag?

Carlos_Bodini · January 15, 2019, 5:25pm

actually not, no error messages

Badger · January 15, 2019, 5:35pm

Enable debug logging. Do you see a line like

[DEBUG][logstash.filters.grok    ] Running grok filter {:event=>#<LogStash::Event:0x4658baf8>}

If not, then your [type] == "iis" still is not matching. But I cannot think of anything else to explain why.

Carlos_Bodini · January 15, 2019, 6:03pm

there's no [DEBUG][logstash.filters.grok ] errors in the log file, it seams the filter is not being triggered by some reason

Rahul07 · January 23, 2019, 4:56pm

Have you got the solution?

Carlos_Bodini · January 23, 2019, 5:08pm

not yet

Rahul07 · January 23, 2019, 7:23pm

Actually i was also facing same issue, with above discussion i got solution.
Can you remove type

input{
file
{ path => "/tmp/***.log"
start_position =>"beginning"
sincedb_path => "NUL"
}
}
filter{
#if[type]=="iis_log"
#{
grok
{
}
}
output
{
elasticsearch
{ hosts => ["localhost:9200"]
index => [""]
# document_type => "iis_log"
}
stdout{}
}

system · February 20, 2019, 7:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Grok fields are not visible in kibana Logstash	9	3383	July 6, 2017
Grok filter not being triggered in Logstash server Logstash	17	1598	March 20, 2019
Filebeat + logstash IIS+kibana Kibana	15	3245	December 18, 2017
Remote IIS logs into ELK Stack? Beats filebeat	5	5532	July 5, 2017
Having trouble with the date filter for iis logs Logstash	4	530	July 9, 2018

Grok filter for IIS no working

Related Topics