IIS Logs - Using Logstash as Forwarder and second Logstash Server indexing the data into Elasticsearch

[vikram_yerneni]

Hello All,
Here is the issue I am facing:
I am using Logstash Agent as a forwarder on our Windows Server 2008 R2 for indexing the IIS logs. And for the logstash forwarder conf file I relied on this site: https://blog.sstorie.com/importing-iis-logs-into-elasticsearch-with-logstash/ and it worked perfectly. I got the IIS logs (W3SVC Logs) into our Elasticsearch Nodes.

Here is our setup: Logstash Forwarder (agent installed on IIS Servers) --> redis --> Logstash Server --> Elasticsearch Data Nodes

However I got an issue with the way how the logs are getting indexed and I want to change the format. The "message" section of the IIS logs consists of multiple items within (as shown below):

"2016-11-22 20:03:48 ::1 GET /test6 - 80 - ::1 Mozilla/5.0+(Windows+NT+6.3;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/54.0.2840.99+Safari/537.36 - 404 0 2 1
"
I am trying to separate these fields as a separate columns. For example, I want to define a datafield for the path from the message "/test6". How can I get that in-place with our current infrastructure? I saw a post for the same if we use the filebeat, but we cant as our redis version is not compatible with the filebeat setups for indexing the IIS logs.

Any help or inputs will be appreciated here folks..

Thanks
Vikram Yerneni

[vikram_yerneni]

Here is the screenshot folks:

[magnusbaeck (Magnus Bäck)]

Add filters in your second Logstash configuration (the one that sends to Elasticsearch). At the very least you'll need a grok and a date filter. There's an example of HTTP log parsing in the Logstash documentation, and while your logging format is slightly different the idea is the same.

[vikram_yerneni]

Thanks Magnusbaeck.. The add the grok filter for the first agent setup and it worked perfectly as the second setup didnt modified the grok filtered queries getting indexed into the Elasticsearch.
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IP:server} %{WORD:method} %{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} (?:%{WORD:username}|-) %{IP:clientip} %{NOTSPACE:useragent} - %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:status} %{NUMBER:timetaken}"]

Thanks
Vikram Yerneni