[IIS 7.5] [Filebeat 6.6.1] Provided Grok expressions do not match field value

Elastic Stack Beats
1

Hello all,

This is giving me headaches. I know there is a lot of posts regarding this issue, but I did not find any real solution.
I'll try to be as comprehensive as possible.

I want to send logs directly to Elastic from my IIS 7.5.
This is my iis.yml configuration :

- module: iis
  # Access logs
  access:
    enabled: true
    var.paths: ["C:/inetpub/logs/LogFiles/*/*.log"]

  # Error logs
  error:
    enabled: true
    var.paths: ["C:/Windows/System32/LogFiles/HTTPERR/*.log"]

on my filebeat.yml, i simply configured the output to elastic :

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["ElasticHost:9200"]

Logs are sent without issue. I can see them in Kibana. BUT.. I have this error (ip modified) :
Provided Grok expressions do not match field value: [2019-03-06 18:43:17 10.0.140.107 GET /health-monitoring - 80 - 10.0.140.2 - 200 0 0 15]

My message field is like this :
t message 2019-03-06 18:43:17 10.0.140.107 GET /health-monitoring - 80 - 10.0.140.2 - 200 0 0 15

And this is my default.json in C:\Program Files\filebeat\module\iis\access\ingest :

{
"description": "Pipeline for parsing IIS access logs. Requires the geoip and user_agent plugins.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:iis.access.server_ip} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.referrer} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.request_time_ms}",
"%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:iis.access.referrer} %{NOTSPACE:iis.access.hostname} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.body_sent.bytes} %{NUMBER:iis.access.body_received.bytes} %{NUMBER:iis.access.request_time_ms}",
"%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{NOTSPACE:iis.access.server_name} %{IPORHOST:iis.access.server_ip} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} HTTP/%{NUMBER:iis.access.http_version} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:iis.access.referrer} %{NOTSPACE:iis.access.hostname} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.body_sent.bytes} %{NUMBER:iis.access.body_received.bytes} %{NUMBER:iis.access.request_time_ms}"
],
"ignore_missing": true
}
}, {
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "iis.access.time",
"target_field": "@timestamp",
"formats": ["yyyy-MM-dd HH:mm:ss"]
}
}, {
"remove": {
"field": "iis.access.time"
}
}, {
"user_agent": {
"field": "iis.access.agent",
"target_field": "iis.access.user_agent"
}
}, {
"rename": {
"field": "iis.access.agent",
"target_field": "iis.access.user_agent.original"
}
}, {
"grok": {
"field": "iis.access.remote_ip",
"patterns": [
"%{NOZONEIP:iis.access.remote_ip_geoip}"
],
"pattern_definitions": {
"NOZONEIP": "[^%]*"
}
}
}, {
"geoip": {
"field": "iis.access.remote_ip_geoip",
"target_field": "iis.access.geoip"
}
}, {
"remove": {
"field": "iis.access.remote_ip_geoip"
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}

And..strange thing, my error logs are correctly parsed!

Sorry if my post was long, this is really annoying, and i need urgently to make it work :smiley:

2

Support for IIS 7.5 has been added in PR 9967. AFAICT this fix will be available in 6.7 and 7.0.

I'm sure you can not wait for the next release. Locate the module folder in your installation. I think with 6.6 it should be safe to just replace module/iis/access/ingest/default.json with the pipeline definition for 6.7: https://raw.githubusercontent.com/elastic/beats/6.7/filebeat/module/iis/access/ingest/default.json

3

Ahhh! Thank you for your answer! Unfortunately, I am still facing the same issue.. I think i'll have to wait for next version :confused:

4

I downloaded the preview version of v7, and still same issue.. :frowning: :crying_cat_face:

5

Hi @nyarlath, I tested it locally and turned out the current filebeat iis module doesn't support 2019-03-06 18:43:17 10.0.140.107 GET /health-monitoring - 80 - 10.0.140.2 - 200 0 0 15 format. The part it doesn't work is IP address 10.0.140.107. Currently we only support [10.0.140.107](http://10.0.140.107). Can you please file an issue in https://github.com/elastic/beats/issues please? Thank you very much!

6

https://github.com/elastic/beats/issues/11255 I just created an issue for this.

7

Thank you for your prompt answer, and thank you for creating this issue.

1 Like
8

Hi,

I believe that I'm experiencing the similar issue that @nyarlath is mentioning.

After being successful with setting up sending iis logs using Filebeat directly to the elasticsearch, on our staging environment, I moved forward with implementation on the production. After following the same steps, I'm seeing in the kibana error message:

Provided Grok expressions do not match field value: [2019-03-20 08:50:21 172.0.0.108 GET /Leads/Lead/FullscreenList - 443 - 176.0.0.198 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/72.0.3626.121+Safari/537.36 ASP.NET_SessionId=4dqnnhh0le4XXitsdbfjm5eg https://inside.com/Leads/Lead/List inside.com 200 0 0 43935 595 588]

I'm running my IIS on 8.5 version and the iis log file has following structure:

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2019-03-20 00:00:07
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2019-03-20 08:50:21 172.0.0.108 GET /Leads/Lead/FullscreenList - 443 - 176.0.0.198 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/72.0.3626.121+Safari/537.36 ASP.NET_SessionId=4dqnnhh0le4XXitsdbfjm5eg https://inside.com/Leads/Lead/List inside.com 200 0 0 43935 595 588

Could you please confirm that the issue that I'm experiencing is the same one mentioned in the thread? If so, how can I fix it?

Thanks for the help!

9

From what I can see, this is the exact same issue I encountered.
To fix this, I sent my logs in logstash, not in Elastic. And I used Dissect instead of Grok.

Find here my Logstash configuration if you want to test it (please note that it is working with IIS 7.5 for me ) :

input {
 beats {
  port => 5044
  #type => "iis"
 }
}

filter {
  dissect {
    mapping => {
      message => '%{log_timestamp} %{+log_timestamp} %{s-sitename} %{s-computername} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs-version} %{cs-user-agent} %{cs-cookie} %{cs-referer} %{cs-host} %{sc-status} %{sc-substatus} %{sc-win32-status} %{sc-bytes} %{cs-bytes} %{time-taken}'
    }
  }
}


output {
 elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-iis-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
 stdout { codec => rubydebug }
10

Thanks for the advice, It worked also in my case!

Although I've managed to import my iis logs into the elasticsearch it was a bit cumbersome and it limited some of the functionalities (i.e. iis.access.geoup.location). @Kaiyan_Sheng can we expect earlier versions of the IIS to be supported in elasticsearch v7?

11

@Jacek https://www.elastic.co/guide/en/beats/filebeat/7.0/filebeat-module-iis.html So far we are supporting iis 7.5 and 10 in filebeat 7.0. If there's a specific earlier version of IIS that you want filebeat to support, you can always create a new issue as Enhancement request in https://github.com/elastic/beats/issues

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.