Using wildcarded filenames in the logstash filter section

Elastic Stack Logstash
1

Is it possible to use wildcarded filenames in the filter plugin section of logstash? For example, I want to index Apache access logs for some of our webapps and since the apache access logs have the date as part of the active filename (localhost_access_log..txt, I am trying to use wildcards. I create index-names based on the source of the log, so in my filter section I have many source entries like:

filter {
if [source] == "/path/to/logfile" {
mutate {
add_field => { 'app_type' => 'some_app_type' }
}
}

This works for all of my sources except the apache access logs with the wildcard.
if [source] == "/path/to/apache/logs/localhost_access_log*.txt" {
mutate {
add_field => { 'app_type' => 'some_app_type' }
}
}

This filter doesnt seem to hit and the Access logs go into a default catchall index. In Kibana I can view the index and the log file entries just fine, even with the source showing up correctly as '/path/to/apache/logs/localhost_access_log..txt

Any ideas on what I may be doing wrong?

2

Is it possible to use wildcarded filenames in the filter plugin section of logstash?

No, but you can perform regular expression matches, see Accessing event data and fields | Logstash Reference [8.11] | Elastic.

3

Thanks! I was able to get it to work with

if [source] == "/path/to/apache/logs/localhost_access_log.%{YYYY-MM-dd}.txt" {.....

4

Ok, maybe I was wrong. Doesnt seem to be working. Is if [source] == "/path/to/apache/logs/localhost_access_log.%{YYYY-MM-dd}.txt" the correct syntax for a regex?

5

No, you need something like

if [source] =~ "/path/to/apache/logs/localhost_access_log.*\.txt$" {

but the best option is to add the desired fields and/or tags at the one place where all of this is known, namely in the input section. You have one input per log type anyway, right?

6

We are using filebeats for the input so I just have that one input:

input {
beats {
port =>"5044"
type => "log"
}
}

In the filter section, based on the app log source I add a field that is then used in the output section for creating the index.

For example:

filter {
if [source] == '/app/log/app1.log' {
mutate {
add_field => { 'app_type' => 'app1'}
}
}

if [source] == '/app/log/app2.log' {
mutate {
add_field => { 'app_type' => 'app2'}
}
}

output {
elasticsearch {
hosts => [ ]
index => "%{app_type}-%{+YYYY.MM.dd}"
}
}

7

We are using filebeats for the input so I just have that one input:

Well, set the field and/or tags on the Filebeat side then. Same thing. The Logstash configuration shouldn't have an exact dependency to the file paths in the Filebeat configuration.

8

I tested adding a field on the Filebeat side

e.g.
fields:
field_name: apache_access

Then on the logstash side under filters

if [field_name] == 'apache_access' {.

Up til now, every single log from beats has a configuration in the filter section in order to apply grok filters and such. I'm sure this is not optimal as the current pipeline is growing large

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.