IIS Logs - Using Logstash as Forwarder and second Logstash Server indexing the data into Elasticsearch Cluster

Hello All,
Here is the issue I am facing:
I am using Logstash Agent as a forwarder on our Windows Server 2008 R2 for indexing the IIS logs. And for the logstash forwarder conf file I relied on this site: https://blog.sstorie.com/importing-iis-logs-into-elasticsearch-with-logstash/ and it worked perfectly. I got the IIS logs (W3SVC Logs) into our Elasticsearch Nodes.

Here is our setup: Logstash Forwarder (agent installed on IIS Servers) --> redis --> Logstash Server --> Elasticsearch Data Nodes

However I got an issue with the way how the logs are getting indexed and I want to change the format. The "message" section of the IIS logs consists of multiple items within (as shown below):

"2016-11-22 20:03:48 ::1 GET /test6 - 80 - ::1 Mozilla/5.0+(Windows+NT+6.3;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/54.0.2840.99+Safari/537.36 - 404 0 2 1
I am trying to separate these fields as a separate columns. For example, I want to define a datafield for the path from the message "/test6". How can I get that in-place with our current infrastructure? I saw a post for the same if we use the filebeat, but we cant as our redis version is not compatible with the filebeat setups for indexing the IIS logs.

Any help or inputs will be appreciated here folks..

Vikram Yerneni

You need to grok the events, basically split them into separate fields - https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

Yes Mark.. I added the below grok and it worked well:

match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IP:server} %{WORD:method} %{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} (?:%{WORD:username}|-) %{IP:clientip} %{NOTSPACE:useragent} - %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:status} %{NUMBER:timetaken}"]

Vikram Yerneni

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.