# IIS logging and \_grokparsefailure tag **URL:** \ **Category:** Logstash\ **Created:** [March 29, 2016, 6:10am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648 "2016-03-29T06:10:06Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 29, 2016, 6:10am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/1 "2016-03-29T06:10:06Z")
Hi, I want to log our Microsoft's IIS logs but getting \_grokparsefailure in tag, though I can see the messages in Kibana but not sure why this tag is being appeared. Here is my NXLOG conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log Module xm\_syslog # # Module xm\_json # # # Module xm\_csv # Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $sc-status, $sc-substatus, $sc-win32-status, $time-taken # FieldTypes string, string, string, string, string, string, integer, string, string, string, integer, integer, integer, integer # Delimiter ' ' # define BASEDIR C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\MessageTracking Module im\_file File "C:\inetpub\logs\LogFiles\W3SVC1\u\_ex\*.log" ReadFromLast True SavePos True Exec if $raw\_event =~ /^#/ drop(); # else \ # { \ # w3c-\>parse\_csv(); \ # $EventTime = parsedate($date + " " + $time); \ # $SourceName = "excahnge\_V1\_iis\_log"; \ # $Message = to\_json(); \ } Module om\_tcp Host 192.195.88.35 Port 3516 Exec $SourceName = 'excahnge\_V1\_iis\_log'; OutputType LineBased \ Path iis\_exchange\_v1 =\> out\_iis\_exchange\_v1 and this is the Logstash: #tcp Exchange IIS logs stream via 3516 tcp { type =\> "iis\_exch\_v1" port =\> 3516 #host =\> "192.195.88.113" } } # end input filter { if [type] == "iis\_exch\_v1" { csv {add\_tag =\> ['exh\_v1\_iis']} grok { match =\> ['@message', "%{TIMESTAMP\_ISO8601:timestamp} %{IPORHOST:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referrer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"] } #if "\_grokparsefailure" in [tags] { # drop { } # } } } output { elasticsearch { hosts =\> ["192.195.88.35:9200"]} #protocol =\> "http" stdout { codec =\> rubydebug} }# end output Could you please help me in this regards if I am doing right? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 29, 2016, 6:14am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/2 "2016-03-29T06:14:17Z")
The `_grokparsefailure` tag indicates a problem with your grok expression. Are you sure you should be matching the expression against the `@message` field rather than `message`? Please show the output of the stdout output so that we can see what the events look like. ---
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 29, 2016, 6:41am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/3 "2016-03-29T06:41:55Z")
This is what you need: @timestamp March 29th 2016, 09:40:28.033 t@version 1 t\_id AVPBGiO3tOFqHDxvrSdW t\_index logstash-2016.03.29 #\_score t\_type iis ?column1 2016-03-29 06:40:27 192.195.88.113 POST /EWS/Exchange.asmx - 443 QATAR\eric.wilson 192.195.88.115 MacOutlook/0.0.0.160212+(Intel+Mac+OS+X+Version+10.11.4+(Build+15E65)) 200 0 0 0 thost 192.195.88.113 tmessage 2016-03-29 06:40:27 192.195.88.113 POST /EWS/Exchange.asmx - 443 QATAR\eric.wilson 192.195.88.115 MacOutlook/0.0.0.160212+(Intel+Mac+OS+X+Version+10.11.4+(Build+15E65)) 200 0 0 0 ?port 42610 ttags exh\_v1\_iis, \_grokparsefailure ttype iis ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 29, 2016, 6:43am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/4 "2016-03-29T06:43:37Z")
No, that's copy/paste from Kibana and nearly unreadable. Please show the output of the stdout output. If you run Logstash as a service it should end up in your log file. When you've verified that things work you should comment out that output to you won't run out of disk space. ---
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 29, 2016, 6:52am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/5 "2016-03-29T06:52:13Z")
Can I attach the log file? Here is the excerpt from it: {:timestamp=\>"2016-03-29T09:47:58.842000+0300", :message=\>"Using mapping template from", :path=\>nil, :level=\>:info} {:timestamp=\>"2016-03-29T09:47:59.061000+0300", :message=\>"Attempting to install template", :manage\_template=\>{"template"=\>"logstash-_", "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "omit\_norms"=\>true}, "dynamic\_templates"=\>[{"message\_field"=\>{"match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}, "fields"=\>{"raw"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true, "ignore\_above"=\>256}}}}}, {"float\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"float", "mapping"=\>{"type"=\>"float", "doc\_values"=\>true}}}, {"double\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"double", "mapping"=\>{"type"=\>"double", "doc\_values"=\>true}}}, {"byte\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"byte", "mapping"=\>{"type"=\>"byte", "doc\_values"=\>true}}}, {"short\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"short", "mapping"=\>{"type"=\>"short", "doc\_values"=\>true}}}, {"integer\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"integer", "mapping"=\>{"type"=\>"integer", "doc\_values"=\>true}}}, {"long\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"long", "mapping"=\>{"type"=\>"long", "doc\_values"=\>true}}}, {"date\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"date", "mapping"=\>{"type"=\>"date", "doc\_values"=\>true}}}, {"geo\_point\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"geo\_point", "mapping"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "doc\_values"=\>true}, "@version"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true}, "geoip"=\>{"type"=\>"object", "dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip", "doc\_values"=\>true}, "location"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}, "latitude"=\>{"type"=\>"float", "doc\_values"=\>true}, "longitude"=\>{"type"=\>"float", "doc\_values"=\>true}}}}}}}, :level=\>:info} {:timestamp=\>"2016-03-29T09:47:59.264000+0300", :message=\>"New Elasticsearch output", :class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["192.195.88.35:9200"], :level=\>:info} {:timestamp=\>"2016-03-29T09:47:59.280000+0300", :message=\>"Using mapping template from", :path=\>nil, :level=\>:info} {:timestamp=\>"2016-03-29T09:47:59.280000+0300", :message=\>"Attempting to install template", :manage\_template=\>{"template"=\>"logstash-_", "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "omit\_norms"=\>true}, "dynamic\_templates"=\>[{"message\_field"=\>{"match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}, "fields"=\>{"raw"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true, "ignore\_above"=\>256}}}}}, {"float\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"float", "mapping"=\>{"type"=\>"float", "doc\_values"=\>true}}}, {"double\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"double", "mapping"=\>{"type"=\>"double", "doc\_values"=\>true}}}, {"byte\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"byte", "mapping"=\>{"type"=\>"byte", "doc\_values"=\>true}}}, {"short\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"short", "mapping"=\>{"type"=\>"short", "doc\_values"=\>true}}}, {"integer\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"integer", "mapping"=\>{"type"=\>"integer", "doc\_values"=\>true}}}, {"long\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"long", "mapping"=\>{"type"=\>"long", "doc\_values"=\>true}}}, {"date\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"date", "mapping"=\>{"type"=\>"date", "doc\_values"=\>true}}}, {"geo\_point\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"geo\_point", "mapping"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "doc\_values"=\>true}, "@version"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true}, "geoip"=\>{"type"=\>"object", "dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip", "doc\_values"=\>true}, "location"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}, "latitude"=\>{"type"=\>"float", "doc\_values"=\>true}, "longitude"=\>{"type"=\>"float", "doc\_values"=\>true}}}}}}}, :level=\>:info} ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 29, 2016, 7:25am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/6 "2016-03-29T07:25:36Z")
No, that's not it. It should look similar to this: ```auto { "message" => "Tue Mar 29 09:25:00 CEST 2016", "@version" => "1", "@timestamp" => "2016-03-29T07:25:07.847Z", "host" => "lnxolofon" } ``` Come to think of it, maybe it ends up in logstash.stdout rather than logstash.log. ---
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 29, 2016, 7:34am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/7 "2016-03-29T07:34:04Z")
Thanks for being helpful Magnus. Should I use stdout { codec =\> rubydebug } or stdout { codec =\> json } as output? ---
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 29, 2016, 7:47am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/8 "2016-03-29T07:47:51Z")
I am running Logstash on Windows Box. I can't find a way to produce stdout output other than log file. Could you please help me in this regard? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 29, 2016, 8:06am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/9 "2016-03-29T08:06:23Z")
> Should I use stdout { codec =\> rubydebug } or stdout { codec =\> json } as output? I think the former gives more readable output. ---
**Author:** ![bhatch](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@bhatch](https://discuss.elastic.co/u/bhatch)\ **Post date:** [March 29, 2016, 11:24pm UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/10 "2016-03-29T23:24:53Z")
The $Message = to\_json(); command in your NXLog is supposed to split up the Message field into separate JSON fields. So by the time it hits Logstash, your data should already be split up. Your GROK expression looks like you are trying to re-split up the message field again. Why is that? At this point it was already split up in NXLog, so you shouldn't need to re-do it again with Grok, you will just end up with duplicate fields. Try commenting out the entire Grok filter, and change your output to file or stdout to confirm what it currently looks like. You may not need the Grok expression at all. ---
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 30, 2016, 5:42am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/11 "2016-03-30T05:42:11Z")
Thanks bhatch... Is there any problem with this module, because it says ERROR invalid keyword: else in log: Module im\_file File "C:\inetpub\logs\LogFiles\W3SVC1\u\_ex\*.log" ReadFromLast True SavePos True ``` Exec if $raw_event =~ /^#/ drop(); \ else\ {\ w3c->parse_csv();\ $SourceName = "exch_v1_iis_log";\ $Message = to_json();\ } ``` ---
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 30, 2016, 5:46am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/12 "2016-03-30T05:46:35Z")
Alright, I fixed it. I had to remove extra spaces in front of back slashes, now if I comment out the whole grok filter, can I still use mutate, date and useragent functions? ---
**Author:** ![bhatch](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@bhatch](https://discuss.elastic.co/u/bhatch)\ **Post date:** [March 30, 2016, 5:32pm UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/13 "2016-03-30T17:32:24Z")
You should be able to. Both the NXLOG parser and your Grok expression give the same end result, it's just a matter of determining which application you want doing it. You can do the CSV to JSON conversion in Logstash by leaving in the Grok expression in and taking it out of NXLog, but we found it faster to have 100 IIS machines do the work in NXLOG than have a handful of Logstash machines do it later down the line. If you were using Logstash File input to read the IISLogs directly, then you would want to use GROK (Or maybe KV) to do the work. We let NXLog do the initial CSV to JSON conversion, and then use various filters such as translate, mutate, useragent, geoip, and grok expressions in Logstash to get the data formatted exactly how we want it. One last thing, the fields names are set in NXLog. It looks like you used a different naming convention from NXLog to Grok. For example, in NXLog you have s-ip, but Grok says hostip. Keep that in mind when creating any further filters. ---
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 31, 2016, 5:15am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/14 "2016-03-31T05:15:12Z")
Hi Bhatch, Thanks a lot for your reply in detail. We, Windows users who are trying to use UNIX based solutions, need this kind of help and guidance. Keep it up. I can see around 12 weird fields in Kibana named like Column1, Column2, Column12, etc. Why these fields are showing up like this? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 31, 2016, 5:30am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/15 "2016-03-31T05:30:34Z")
> I can see around 12 weird fields in Kibana named like Column1, Column2, Column12, etc. Why these fields are showing up like this? Because you use a csv filter without naming the columns? ---
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 31, 2016, 10:57am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/16 "2016-03-31T10:57:26Z")
Hi: This is my nxlog.conf: ``` Module xm_csv Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - Module im_file File "C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log" ReadFromLast True SavePos True Exec if $raw_event =~ /^#/ drop();\ else\ {\ w3c->parse_csv();\ $SourceName = "exch_v2_iis_log";\ $Message = to_json();\ } ``` How and where should I name the columns? Sorry for newbie question. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 31, 2016, 11:07am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/17 "2016-03-31T11:07:46Z")
I was assuming that it was your csv filter in Logstash that was doing the filtering. If the CSV parsing happens on the NXLog side I can't help. ---
**Author:** ![thyfere](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thyfere](https://discuss.elastic.co/u/thyfere)\ **Post date:** [March 31, 2016, 11:59am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/18 "2016-03-31T11:59:08Z")
Thanks it's fixed. Though, I was using only this `csv {add_tag => ['exh_iis_log']}`. As soon as I commented it, columns disappeared. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:04am UTC](https://discuss.elastic.co/t/iis-logging-and--grokparsefailure-tag/45648/19 "2017-07-06T05:04:31Z")