Logstash Grok Filter IIS parsing to ElasticSearch

Elastic Stack Logstash
1

Dear Comunity,

I explain the situation, I have Filebeat installed in a server where there is an IIS configured , I retrieve the logs and send them to a logstash server which will output to a elasticsearch node.

This is working but now I want to use grok in order to have separate fields in my index.

Here an example of data from the IIS logs

 #cat data
2018-02-08 00:00:08 192.168.1.36 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Sync&User=test%test&DeviceId=androidc1393388699&DeviceType=BlackBerry&Log=V141_Fc7_Fid:6_St:S_Sk:1733558382_Fid:3_Ty:Co_Filt0_St:S_Sk:746929482_Sst32_BR1_BPR0_Fid:1_Ty:Ca_Filt4_St:S_Sk:375762110_Sst36_BR1_BPR0_Fid:54_St:S_Sk:1964429127_Fid:5_St:S_Sk:789689042_Fid:56_St:S_Sk:1204425375_Fid:46_St:S_Sk:630047969_LdapC1_RpcC80_RpcL31_Hb540_Rto1_Ers1_Cpo9949_Fet540005_Pk1365808096_Mbx:GEOSRVEXCH36.Geogas.local_Throttle0_Budget:(D)Conn%3a1%2cHangingConn%3a0%2cAD%3a%24null%2f%24null%2f0%25%2cCAS%3a%24null%2f%24null%2f1%25%2cAB%3a%24null%2f%24null%2f0%25%2cRPC%3a%24null%2f%24null%2f0%25%2cFC%3a1000%2f0%2cPolicy%3aDefaultThrottlingPolicy%5F0e9bbb7d-3889-4cfa-b214-7ad1b2b7fe06%2cNorm_ 443 geogas\raickman 1.1.1.1 BlackBerry-BBA100-2/6.0.1 200 0 0 540802

Here my filter

 #cat test.config
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
        grok {

                match => {"message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{IP:s-ip} %{URIPROTO:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IP:c-ip} %{NOTSPACE:csuser-agent} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken}"}
        }
}

When i try to check if logstash is parsing it it says ok

# /usr/share/logstash/bin/logstash -f test.config < data
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
{
    "sc-win32-status" => "0",
          "cs-method" => "POST",
        "cs-username" => "test\\test",
       "sc-substatus" => "0",
         "@timestamp" => 2018-02-08T14:17:54.866Z,
       "cs-uri-query" => "Cmd=Sync&User=test%test&DeviceId=androidc1393388699&DeviceType=BlackBerry&Log=V141_Fc7_Fid:6_St:S_Sk:1733558382_Fid:3_Ty:Co_Filt0_St:S_Sk:746929482_Sst32_BR1_BPR0_Fid:1_Ty:Ca_Filt4_St:S_Sk:375762110_Sst36_BR1_BPR0_Fid:54_St:S_Sk:1964429127_Fid:5_St:S_Sk:789689042_Fid:56_St:S_Sk:1204425375_Fid:46_St:S_Sk:630047969_LdapC1_RpcC80_RpcL31_Hb540_Rto1_Ers1_Cpo9949_Fet540005_Pk1365808096_Mbx:GEOSRVEXCH36.Geogas.local_Throttle0_Budget:(D)Conn%3a1%2cHangingConn%3a0%2cAD%3a%24null%2f%24null%2f0%25%2cCAS%3a%24null%2f%24null%2f1%25%2cAB%3a%24null%2f%24null%2f0%25%2cRPC%3a%24null%2f%24null%2f0%25%2cFC%3a1000%2f0%2cPolicy%3aDefaultThrottlingPolicy%5F0e9bbb7d-3889-4cfa-b214-7ad1b2b7fe06%2cNorm_",
      "log_timestamp" => "2018-02-08 00:00:08",
       "csuser-agent" => "BlackBerry-BBA100-2/6.0.1",
          "sc-status" => "200",
               "s-ip" => "192.168.1.36",
             "s-port" => "443",
         "time-taken" => "540802",
           "@version" => "1",
               "host" => "server.domain.local",
            "message" => "2018-02-08 00:00:08 192.168.1.36 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Sync&User=test%5test&DeviceId=androidc1393388699&DeviceType=BlackBerry&Log=V141_Fc7_Fid:6_St:S_Sk:1733558382_Fid:3_Ty:Co_Filt0_St:S_Sk:746929482_Sst32_BR1_BPR0_Fid:1_Ty:Ca_Filt4_St:S_Sk:375762110_Sst36_BR1_BPR0_Fid:54_St:S_Sk:1964429127_Fid:5_St:S_Sk:789689042_Fid:56_St:S_Sk:1204425375_Fid:46_St:S_Sk:630047969_LdapC1_RpcC80_RpcL31_Hb540_Rto1_Ers1_Cpo9949_Fet540005_Pk1365808096_Mbx:GEOSRVEXCH36.Geogas.local_Throttle0_Budget:(D)Conn%3a1%2cHangingConn%3a0%2cAD%3a%24null%2f%24null%2f0%25%2cCAS%3a%24null%2f%24null%2f1%25%2cAB%3a%24null%2f%24null%2f0%25%2cRPC%3a%24null%2f%24null%2f0%25%2cFC%3a1000%2f0%2cPolicy%3aDefaultThrottlingPolicy%5F0e9bbb7d-3889-4cfa-b214-7ad1b2b7fe06%2cNorm_ 443 test\\test 1.1.1.1 BlackBerry-BBA100-2/6.0.1 200 0 0 540802",
        "cs-uri-stem" => "/Microsoft-Server-ActiveSync/default.eas",
               "c-ip" => "1.1.1.1"
}

Here my config from logstash

input {
        beats {
                port => 5044
        }
}
filter {

        if [type] == "iis" {
                if [message] =~ "^#" {
                        drop {}
                }
                grok {
                        match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{IP:s-ip} %{URIPROTO:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IP:c-ip} %{NOTSPACE:csuser-agent} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken}" }

                }
        }

}
output {

        elasticsearch {
                hosts => ["http://192.168.1.32:9200"]
                index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }
}

But the problem is that i dont have this field in elasticsearch , the index is not updating with the new fields

Do you have an idea why ?

Best Regards, Edouard Fazenda.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.