IIS Logs - Using Logstash as Forwarder and second Logstash Server indexing the data into Elasticsearch Cluster

[vikram_yerneni]

Hello All,
Here is the issue I am facing:
I am using Logstash Agent as a forwarder on our Windows Server 2008 R2 for indexing the IIS logs. And for the logstash forwarder conf file I relied on this site: https://blog.sstorie.com/importing-iis-logs-into-elasticsearch-with-logstash/ and it worked perfectly. I got the IIS logs (W3SVC Logs) into our Elasticsearch Nodes.

Here is our setup: Logstash Forwarder (agent installed on IIS Servers) --> redis --> Logstash Server --> Elasticsearch Data Nodes

However I got an issue with the way how the logs are getting indexed and I want to change the format. The "message" section of the IIS logs consists of multiple items within (as shown below):

"2016-11-22 20:03:48 ::1 GET /test6 - 80 - ::1 Mozilla/5.0+(Windows+NT+6.3;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/54.0.2840.99+Safari/537.36 - 404 0 2 1
"
I am trying to separate these fields as a separate columns. For example, I want to define a datafield for the path from the message "/test6". How can I get that in-place with our current infrastructure? I saw a post for the same if we use the filebeat, but we cant as our redis version is not compatible with the filebeat setups for indexing the IIS logs.

Any help or inputs will be appreciated here folks..

Thanks
Vikram Yerneni

[warkolm (Mark Walkom)]

You need to grok the events, basically split them into separate fields - https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

[vikram_yerneni]

Yes Mark.. I added the below grok and it worked well:

match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IP:server} %{WORD:method} %{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} (?:%{WORD:username}|-) %{IP:clientip} %{NOTSPACE:useragent} - %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:status} %{NUMBER:timetaken}"]

Thanks
Vikram Yerneni