Using wildcarded filenames in the logstash filter section

Is it possible to use wildcarded filenames in the filter plugin section of logstash? For example, I want to index Apache access logs for some of our webapps and since the apache access logs have the date as part of the active filename (localhost_access_log..txt, I am trying to use wildcards. I create index-names based on the source of the log, so in my filter section I have many source entries like:

filter {
if [source] == "/path/to/logfile" {
mutate {
add_field => { 'app_type' => 'some_app_type' }
}
}

This works for all of my sources except the apache access logs with the wildcard.
if [source] == "/path/to/apache/logs/localhost_access_log*.txt" {
mutate {
add_field => { 'app_type' => 'some_app_type' }
}
}

This filter doesnt seem to hit and the Access logs go into a default catchall index. In Kibana I can view the index and the log file entries just fine, even with the source showing up correctly as '/path/to/apache/logs/localhost_access_log..txt

Any ideas on what I may be doing wrong?

Is it possible to use wildcarded filenames in the filter plugin section of logstash?

No, but you can perform regular expression matches, see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html.

Thanks! I was able to get it to work with

if [source] == "/path/to/apache/logs/localhost_access_log.%{YYYY-MM-dd}.txt" {.....

Ok, maybe I was wrong. Doesnt seem to be working. Is if [source] == "/path/to/apache/logs/localhost_access_log.%{YYYY-MM-dd}.txt" the correct syntax for a regex?

No, you need something like

if [source] =~ "/path/to/apache/logs/localhost_access_log.*\.txt$" {

but the best option is to add the desired fields and/or tags at the one place where all of this is known, namely in the input section. You have one input per log type anyway, right?

We are using filebeats for the input so I just have that one input:

input {
beats {
port =>"5044"
type => "log"
}
}

In the filter section, based on the app log source I add a field that is then used in the output section for creating the index.

For example:

filter {
if [source] == '/app/log/app1.log' {
mutate {
add_field => { 'app_type' => 'app1'}
}
}

if [source] == '/app/log/app2.log' {
mutate {
add_field => { 'app_type' => 'app2'}
}
}

output {
elasticsearch {
hosts => [ ]
index => "%{app_type}-%{+YYYY.MM.dd}"
}
}

We are using filebeats for the input so I just have that one input:

Well, set the field and/or tags on the Filebeat side then. Same thing. The Logstash configuration shouldn't have an exact dependency to the file paths in the Filebeat configuration.

I tested adding a field on the Filebeat side

e.g.
fields:
field_name: apache_access

Then on the logstash side under filters

if [field_name] == 'apache_access' {.

Up til now, every single log from beats has a configuration in the filter section in order to apply grok filters and such. I'm sure this is not optimal as the current pipeline is growing large

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.