Great! Below is my final code. I decided for any logs under a sub-directory to determine the type
based on that directory name. For example /var/log/logsfrommanyapps/app1/app1-test.log
is of type => app1
(using the grok filter that you show in your first response). Then for any logs in the base directory, I am using the name of the log file. E.g. /var/log/logsfrommanyapps/test.log
is of type => test
as you show above. However, there is an issue in displaying the type
for logs under a sub-directory. For /var/log/logsfrommanyapps/app1/app1-test.log
it shows as type => app1/app1-test
instead of type => app1
.
Edit: to avoid any further problems for any logs under a subdirectory, is it possible to just give it the second level directory name to type
. For example /var/log/logsfrommanyapps/app1/anotherdirectory/app1-test.log
will give type => app1
.
input{
file {
path => ["/var/log/logsfrommanyapps/*/*.log"]
ignore_older => 7776000
start_position => "beginning"
sincedb_path => "/dev/null"
}
file {
path => ["/var/log/logsfrommanyapps/*.log"]
ignore_older => 7776000
start_position => "beginning"
sincedb_path => "/dev/null"
}
filter {
grok {
match => { "path" => [
"^/var/log/logsfrommanyapps/%{WORD:type}/%{DATA}$"
"^/var/log/logsfrommanyapps/%{NOTSPACE:type}.log$"
]}
overwrite => [ "type" ]
}
}
output {
stdout { codec => rubydebug { metadata => true } }
}
}