How to set different filters for different logs in beats and logstash 6.3.2

[Anil_Bind]

Hi,

First of all, I am using version 6.3.2 for all the beats and elk stack.
I have elk setup on centOS 7 and the node is windows server 2012 r2, I was able to filter IIS logs by using below filter.

filter {

if [@metadata][beat] == "filebeat" {
{
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} (?:-|"%{URIPATH:CS-URI-Query}") %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
}
}
}
}

The issue is I have a path having custom logs, I have grok pattern also for it, but IDK how to add that filter with the above existing filter
In the filebeat.yml I have added the path of the log but it uses the pattern of the iss.
ANY HELP!

PS. I have not used iis module because it does not works with iis 8.5.

[kvch (Noémi Ványi)]

To parse your custom logs you need to add a new grok filter after the existing one.

if [@metadata][beat] == "filebeat" {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} {NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} (?:-|"%{URIPATH:CS-URI-Query}") %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
    }
    grok {
       match => { "message" => "{{ your-custom-pattern }}"}
    }
}

Alternatively, you could extend the existing pipeline of IIS module of Filebeat. You need to edit module/iis/access/ingest/default.json or module/iis/error/ingest/default.json depending on which fileset you need. A new pattern can be added to the list of processors/grok/patterns, so Ingest node can match your logs. But this requires you to forward events to Elasticsearch.

Feel free to open a pull request on Github with the pattern. It would be appreciated if you contributed it for IIS 8.5.

[Anil_Bind]

Thanks for ur help, will contribute in GitHub too.