Issue with GROK parsing of IIS NXLOG data

Elastic Stack Logstash
1

I'm using NXLOG to send my IIS logs to my ELK server. Everything is fine except for one part of the message. The port identified in the IIS Message is getting changed. See Bellow

Logstash Filter:

filter {
if [type] == "iis" {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:timeofevent} %{IPORHOST:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:cs-host} %{NUMBER:status} %{NUMBER:response} %{NUMBER:win32status} %{NUMBER:timetaken}"}
}
date {
match => [ "timeofevent", "YYYY-MM-dd HH:mm:ss" ]
target => "@timeofevent"
timezone => "America/Los_Angeles"
}
}
}

Logstash Jason data

{
"_index": "iis",
"_type": "main",
"_id": "AVehTvilI-4o-72swUEx",
"_score": null,
"_source": {
"message": "2016-10-07 22:41:01 x.x.x.27 POST /authStatus/authStatus.aspx - 443 766 x.x.x.6 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko x.somedomain.com 200 0 0 16765\r",
"@version": "1",
"@timestamp": "2016-10-07T22:41:28.999Z",
"host": "X.X.X.28",
"port": 51096,
"type": "iis",
"timeofevent": "2016-10-07 22:41:01",
"hostip": "X.X.X.27",
"method": "POST",
"page": "/authStatus/authStatus.aspx",
"query": "-",
"username": "766",
"clientip": "X.X.X.6",
"useragent": "Mozilla/5.0+(Windows+NT+6.1;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko",
"cs-host": "x.somedomain.com",
"status": "200",
"response": "0",
"win32status": "0",
"timetaken": "16765",
"@timeofevent": "2016-10-08T05:41:01.000Z"
},
"fields": {
"timeofevent": [
1475880061000
],
"@timestamp": [
1475880088999
]
},
"sort": [
1475880061000
]
}

It looks like "port": 51096, matches the client port for the NXLOG client/windows IIS server, but im not sure if thats just a coincidence or how it would get that info, if its only suppose to be parsing the message it receives.

any help with this would be great.

2

Since the 'port' field already exists from the NXLog message itself, you need to tell the Grok filter to allow it to overwrite the value in that field.

filter {
  if [type] == "iis" {
    grok {
      match => {"message" => "%{TIMESTAMP_ISO8601:timeofevent} %{IPORHOST:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:cs-host} %{NUMBER:status} %{NUMBER:response} %{NUMBER:win32status} %{NUMBER:timetaken}"}
      overwrite => [ "port" ]
    }
    date {
      match => [ "timeofevent", "YYYY-MM-dd HH:mm:ss" ]
      target => "@timeofevent"
      timezone => "America/Los_Angeles"
    }
  }
}

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-overwrite

As a test, I would remove that filter and just take a look at all the information that NXlog is sending over, there is a few fields by default in addition to the 'message' field.

1 Like
3

thanks Jared, "overwrite" worked like a charm.

Manny