How to apply different filters based on particular name of files without using Filebeat

Elastic Stack Logstash
1

I want to perform a unique grok filter per log. So let's say for 1.log I want to perform filter A, 2.log I want filter B, and so on. I've copied my input section in the logstash configuration file with psuedo code in the filter section that I am asking about. Thanks in advance!

input {
       file {
		path => "/home/joseph/logs/1.log"
		path => "/home/joseph/logs/2.log"
		path => "/home/joseph/logs/3.log"
		path => "/home/joseph/logs/4.log"
        // in reality, we have path => "/home/joseph/logs/*"
       start_position => "beginning"
	}
}

filter {
	//if name of file is 1.log {
		// filter A
	}
	//if name of file is 2.log {
		// filter B
	}
	//if name of file is 3.log {
		// filter C
	}
	//if name of file is 4.log {
		// filter D
	}
}
2

The file input adds a path field to every event, so you could test

if [path] == "/home/joseph/logs/1.log"

or even

if [path] =~ /\/1.log$/
3

Thank you! So I can use the grok match filter on the path as well? It doesn't seem to be pattern matching correctly for me. Even when I simply match "path" to %{GREEDYDATA}, I am getting a _grokparsefailure

filter {
        grok {
                match => {
                       "path" => "%{GREEDYDATA}%{SYSID:sysid}%{KEYWORD:keyword}"     
                }
                pattern_definitions => {
                                 "SYSID" => "[\w]{1}"
                                "KEYWORD" => ".log"

                }
4

With [path] set to "/home/joseph/logs/1.log" that grok works just fine for me. However, I would write it differently. There is no need for the initial %{GREEDYDATA} since the pattern is not anchored to the start of the field. Also, I would anchor it to the end of the field. Also, \w is already a character class, so there is no need for [ and ] around it.

    grok {
        match => { "path" => "%{SYSID:sysid}\.log$" }
        pattern_definitions => { "SYSID" => "\w{1}" }
    }

But regardless of the stylistic points, your grok works just fine for me, so I am not sure what the issue is.

5

Thank you for the cleaner code! I'm definitely still curious why it isn't working on my end...
Update: I needed the $ at the end of the line

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.