Hi, I have to first apologize for my ignorance as i JUST configured my ELK setup and am getting familiar with the technology, Here is my issue... I would like to ship logs from a dozen or so IIS servers to my Log server which is utilizing Elasticsearch, Logstash and Kibana. I can't seem to find the right documentation on how to get those IIS logs visible and searchable in Kibana. I understand that I'll need to use filebeat but unsure of what additional configuration is needed to get it to work appropriately. Any assistance would be greatly appreciated!
I would recommend following the Getting Started guide for Filebeat. First ship your logs directly to Elasticsearch for learning purposes. Then add Logstash to the mix (Filebeat -> Logstash -> Elasticsearch) to do some grok parsing on your log messages.
I have it pulling logs locally without issue. But I seem to be having issues when attempting to get the IIS logs from my other web servers. I'd be happy to show config files if that'd help. I'm so new to this that I'm unsure at this point what would help troubleshooting. Thanks!
My issue is I have a filebeats input on my elk stack. I have a syslog filter which works great. I setup an IIS filter. The pattern passes the grok filter when I test via https://grokdebug.herokuapp.com/, however, when I attempt to pull the data in Kibana, I'm getting issues with the grok parsing. I can see the fields in the kibana filebeat settings, so they seem to be imported correctly. Here is a sample of my "iis" filter:
filter {
if [type] == "iis" {
grok {
pattern => "%{DATESTAMP:EventTime} %{WORD:sitename} %{HOSTNAME:computername} %{IP:hostip} %{URIPROTO:method} %{URIPATH:request} (?:%{NOTSPACE:queryparam}|-) %{NUMBER:port} (?:%{WORD:username}|-) %{IP:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:user-agent} (?:%{NOTSPACE:cookie}|-) (?:%{NOTSPACE:referer}|-) (?:%{HOSTNAME:host}|-) %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"
}
geoip {
source => "clientip"
target => "geoip"
add_tag => ["iis_geoip"]
}
date {
match=> ["EventTime", "yyyy-MM-dd HH:mm:ss"]
}
}
}