How filebeat collect iis log through grok filter? tags:_grokparsefailure

Elastic Stack Logstash
1

I want to use logstash grok collect iis_log,but I didn't succeed.[tags:_grokparsefailure]

my iis_log format:

2016-01-25 07:54:53 11.11.120.130 GET /Portal/Content/HtmlImageHandler.ashx fid=50ffc09402534d70b17736b7b307475a 80 - 11.112.11.83 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.63+Safari/537.36+SE+2.X+MetaSr+1.0 200 0 0 38

Or this format:

2016-01-25 07:55:50 11.14.13.130 GET /A4Log/Rest/Query - 80 - 11.9.10.39 - 200 0 0 2

my iis patterns format:

IISLOG %{DATE_EN:date} %{TIME:time} %{IP:server_ip} %{WORD:request_type} %{URIPATHPARAM:request} (?:%{URIPATHPARAM:request_}|-) %{NUMBER:port} - %{IP:client_ip} Mozilla%{URIPATHPARAM:agent} %{NUMBER:response} %{NUMBER:bytes} %{NUMBER:win32_status} %{NUMBER:time_taken}

my logstash grok :
else if [fields][tgs] == "iis_log" {
grok {
match => { "message" => "%{IISLOG}" }
}
}

But result is [tags:_grokparsefailure]， i'dont konw where are wrong ，please help me，I just want to have a good year in China！:monkey:

2

I'm not sure DATE_EN is even a valid pattern, but even if it is I suspect it's the wrong pattern to use for yyyy-mm-dd dates. I suggest you use TIMESTAMP_ISO8601 to match both the date and the time.

If things still don't work after making that change, start with the simplest possible expression,

^{TIMESTAMP_ISO8601:timestamp}

and make sure that works. Then add the next token,

^{TIMESTAMP_ISO8601:timestamp} %{IP:server_ip}

and the next, and so on until things break. Then you know what part of the expression is bad.

3

Thank you very much, you let me solve my problem.Thanks again.

But I now have a new problem, when I use the following format, can fully match iis_log:

else if [fields][tgs] == "iis_log" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:datetime} %{IP:server_ip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IP:client_ip} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}" }
}
}

The results of:datetime:2016-01-25 08:52:14 method:POST page:/A4Log/Rest/Save query:- port:80 ....

But when I use the following format, the match would be prompted to fail:

else if [fields][tgs] == "iis_log" {
grok {
match => { "message" => "%{IISLOG}" }
}
}
The results of:tags:_grokparsefailure

Is my patterns file path：
[root@ESK01 patterns]# pwd
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.2/patterns
[root@ESK01 patterns]# ls -lh
total 104K
......
-rw-r--r--. 1 logstash logstash 6.2K Jan 25 17:13 grok-patterns
......
-rw-r--r--. 1 logstash logstash 85 Jan 25 17:23 iis
[root@ESK01 patterns]# cat iis
IISLOG %{TIMESTAMP_ISO8601:datetime} %{IP:server_ip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IP:client_ip} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}

Can't use "% {IISLOG}" this model?

4

I'm not sure what's wrong here. If you start Logstash with --verbose or --debug it'll list exactly which pattern files and which patterns are loaded. Perhaps that'll give you some clues.

5

Thank you very much for your help
Use % {IISLOG} He has already started to work.
But I did not do what I don't know much about it This is the reason why.
Overall my problem has been solved.Thank you very much for your know.thank you

{
  "_index": "iis_log-2016.01.26",
  "_type": "log",
  "_id": "AVJ7-van_KhNv7U12VXl",
  "_score": null,
  "_source": {
    "@version": "1",
    "@timestamp": "2016-01-26T03:29:47.431Z",
    "beat": {
      "hostname": "VWEB30",
      "name": "VWEB30"
    },
    "count": 1,
    "fields": {
      "tgs": "iis_log"
    },
    "input_type": "log",
    "offset": 8444361,
    "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex160126.log",
    "type": "log",
    "host": "VWEB30",
    "datetime": "2016-01-26 03:28:55",
    "server_ip": "11.1.1.30",
    "method": "GET",
    "page": "/Portal/Content/Handler.ashx",
    "query": "fid=48ddc978891c4207878db3b149466e24",
    "port": "80",
    "username": "-",
    "client_ip": "12.1.1.8",
    "useragent": "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1)",
    "response": "200",
    "subresponse": "0",
    "scstatus": "0",
    "timetaken": "30"
  },
  "fields": {
    "@timestamp": [
      1453778987431
    ]
  },
 "sort": [
    1453778987431
  ]
}