Parsing iis log

Elastic Stack Logstash
1

I am new to grok function, can someone you help me to work this out
my log look like :

#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2018-04-25 00:00:00
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2018-04-25 00:00:00 100.00.0.00 HEAD / - 80 - 00.00.0.0 WhatsUp/0.0 - 1 1 1 62

and my conf file look like

input {
beats {
port => 5044
type => "log"
}
}
filter {
mutate {
add_field => {
"log_timestamp" => "%{date} %{time}"
}
}
}

grok {
match => {"message" => "%{TIMESTAMP_ISO8601:log_timestamp}%{SPACE}%{IPORHOST:s-ip}%{SPACE}%{WORD:cs-method}%{SPACE}%{URIPATH:cs-uri-stem}%{SPACE}%{IPORHOST:c-ip}%{SPACE}%{NOTSPACE:cs-(User-Agent)}%{SPACE}%{WORD:sc(Referer)}%{SPACE}%{NUMBER:sc-status}%{SPACE}%{NUMBER:sc-substatus}%{SPACE}%{NUMBER:sc-win32-status}%{SPACE}%{NUMBER:time-taken}"
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "iis_test1-%{+YYYY.MM}"
document_type => "log"
}
}

the massage do't parsed, I'm getting one field
t message 2018-05-01 23:58:49 172.18.1.23 GET /F5/F5.htm - 443 - 00.04.10.00 - - 200 0 0 0

2

You can use Dissect, its better suited to log lines of very regular shape.

Notes:
Try not to use field names like cs-(User-Agent) with parentheses.
I simply took the #Fields line and put %{ and } on either side of the field. I replaced the cs-(User-Agent) with cs-user-agent and used %{log_timestamp} %{+log_timestamp} instead of date and time.

input {
  generator {
    lines => [
      "2018-04-25 00:00:00 100.00.0.00 HEAD / - 80 - 00.00.0.0 WhatsUp/0.0 - 1 1 1 62",
      "2018-05-01 23:58:49 172.18.1.23 GET /F5/F5.htm - 443 - 00.04.10.00 - - 200 0 0 0"
    ]
    count => 1
  }
}

filter {
  dissect {
    mapping => {
      message => '%{log_timestamp} %{+log_timestamp} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs-user-agent} %{cs-referer} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}'
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Gives

{
             "s-port" => "443",
       "cs-uri-query" => "-",
    "sc-win32-status" => "0",
            "message" => "2018-05-01 23:58:49 172.18.1.23 GET /F5/F5.htm - 443 - 00.04.10.00 - - 200 0 0 0",
               "c-ip" => "00.04.10.00",
               "s-ip" => "172.18.1.23",
        "cs-uri-stem" => "/F5/F5.htm",
        "cs-username" => "-",
           "@version" => "1",
         "cs-referer" => "-",
           "sequence" => 0,
               "host" => "Elastics-MacBook-Pro.local",
      "cs-user-agent" => "-",
       "sc-substatus" => "0",
         "time-taken" => "0",
      "log_timestamp" => "2018-05-01 23:58:49",
          "cs-method" => "GET",
         "@timestamp" => 2018-05-29T20:25:17.199Z,
          "sc-status" => "200"
}
{
             "s-port" => "80",
       "cs-uri-query" => "-",
    "sc-win32-status" => "1",
            "message" => "2018-04-25 00:00:00 100.00.0.00 HEAD / - 80 - 00.00.0.0 WhatsUp/0.0 - 1 1 1 62",
               "c-ip" => "00.00.0.0",
               "s-ip" => "100.00.0.00",
        "cs-uri-stem" => "/",
        "cs-username" => "-",
           "@version" => "1",
         "cs-referer" => "-",
           "sequence" => 0,
               "host" => "Elastics-MacBook-Pro.local",
      "cs-user-agent" => "WhatsUp/0.0",
       "sc-substatus" => "1",
         "time-taken" => "62",
      "log_timestamp" => "2018-04-25 00:00:00",
          "cs-method" => "HEAD",
         "@timestamp" => 2018-05-29T20:25:17.174Z,
          "sc-status" => "1"
}
1 Like
3

@guyboertje, thank you so mach, it work great.

4

Make sure you check any previous logs for a cs-uri-query value that is not Percent encoded. If a space leaks into that value the mapping will not give the correct results. e.g. should be foo%20bar and not foo bar.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.