Adding a type field dynamically for logs

Elastic Stack Logstash
1

Logstash is gathering logs from multiple directories. Each directory represents logs from an specific application. I am using the type to indicate the type of logs. However, I have many applications and adding each one with its correponding type to the configuration file is becoming quite overwhelming. Is there a way to this dynamically?(similar to shown below). The type is determined by the name of the log file.

Current:

    file {
            path => ["/var/log/logsfrommanyapps/app1/app1.log"]
			type => "app1",
            ignore_older => 7776000
            start_position => "beginning"
            sincedb_path => "/dev/null"
    }
	file {
            path => ["/var/log/logsfrommanyapps/app2/app2.log"]
			type => "app2",
            ignore_older => 7776000
            start_position => "beginning"
            sincedb_path => "/dev/null"
    }
	file {
            path => ["/var/log/logsfrommanyapps/app3/app3.log"]
			type => "app3",
            ignore_older => 7776000
            start_position => "beginning"
            sincedb_path => "/dev/null"
    }
	output {
		stdout { codec => rubydebug { metadata => true } }
	}

Desired way dynamically:

    file {
            path => ["/var/log/logsfrommanyapps/**/*.log"]
			type => "%{type}"
            ignore_older => 7776000
            start_position => "beginning"
            sincedb_path => "/dev/null"
    }
	output {
		stdout { codec => rubydebug { metadata => true } }
	}
2

You can use grok filter to do that :

input {
    file {
            path => ["/var/log/logsfrommanyapps/*/*.log"]
            ignore_older => 7776000
            start_position => "beginning"
            sincedb_path => "/dev/null"
    }
}

filter {
    grok {
        match => { "path" => "/var/log/logsfrommanyapps/%{WORD:type}/%{DATA}" }
        overwrite => [ "type" ]
    }
}

output {
		stdout { codec => rubydebug { metadata => true } }
}
3

Awesome that works great! One additional question in relation to this. Logstash adds a host field name. Currently it shows e.g. "host" => "servername.mydomain.com". How would I be able to parse that field so it only outputs "host" => "servername"?

4

It is the same idea:

filter {
grok {
match => { "host" => "^%{WORD:host}." }
overwrite => [ "host" ]
}
}

5

Ah perfect, I was close, thanks. Ran into an issue with the first grok for the type. I have the following log "/var/log/logsfrommanyapps/app3/app3-access.log"with the current grok it shows type type => "app3" but would like for the type to be the filename e.g. type => "app3-access"

6

filter {
grok {
match => { "path" => "^/var/log/logsfrommanyapps/%{WORD}/%{NOTSPACE:type}.log$" }
overwrite => [ "type" ]
}
}

7

Sorry to bother again and this will be the last. I realized I also have logs in the base directory /var/log/logsfrommanyapps/*.log the grok parse works well for settingtype for logs coming from the subdirectories: /var/log/logsfrommanyapps/*/*.log but not from the base directory /var/log/logsfrommanyapps/*.log. How to grok for such logs?

Below is my updated configuration with your previous answer:

input{
    file {
            path => ["/var/log/logsfrommanyapps/*/*.log"]
            ignore_older => 7776000
            start_position => "beginning"
            sincedb_path => "/dev/null"
    }
    file {
            path => ["/var/log/logsfrommanyapps/*.log"]
			type => "%{type}"
            ignore_older => 7776000
            start_position => "beginning"
            sincedb_path => "/dev/null"
    }
    filter {
       grok {
          match => { "path" => "^/var/log/logsfrommanyapps/%{WORD}/%{NOTSPACE:type}.log$"}
          overwrite => [ "type" ]
       }
    }
	output {
		stdout { codec => rubydebug { metadata => true } }
	}
}
8

You could do that using the following configuration. If first pattern is not matched, the second is used.

filter {
  grok {
    match => { "path" => [
      "^/var/log/logsfrommanyapps/%{WORD}/%{NOTSPACE:type}.log$" 
      "^/var/log/logsfrommanyapps/%{NOTSPACE:type}.log$" 
    ]}
    overwrite => [ "type" ]
  }
}

One last thing : it is useless to set type => "%{type}" in file input.

9

Great! Below is my final code. I decided for any logs under a sub-directory to determine the type based on that directory name. For example /var/log/logsfrommanyapps/app1/app1-test.log is of type => app1 (using the grok filter that you show in your first response). Then for any logs in the base directory, I am using the name of the log file. E.g. /var/log/logsfrommanyapps/test.log is of type => test as you show above. However, there is an issue in displaying the type for logs under a sub-directory. For /var/log/logsfrommanyapps/app1/app1-test.log it shows as type => app1/app1-test instead of type => app1.

Edit: to avoid any further problems for any logs under a subdirectory, is it possible to just give it the second level directory name to type. For example /var/log/logsfrommanyapps/app1/anotherdirectory/app1-test.log will give type => app1.

input{
    file {
            path => ["/var/log/logsfrommanyapps/*/*.log"]
            ignore_older => 7776000
            start_position => "beginning"
            sincedb_path => "/dev/null"
    }
    file {
            path => ["/var/log/logsfrommanyapps/*.log"]
            ignore_older => 7776000
            start_position => "beginning"
            sincedb_path => "/dev/null"
    }
    filter {
       grok {
              match => { "path" => [
                  "^/var/log/logsfrommanyapps/%{WORD:type}/%{DATA}$" 
                  "^/var/log/logsfrommanyapps/%{NOTSPACE:type}.log$" 
           ]}
          overwrite => [ "type" ]
       }
    }
	output {
		stdout { codec => rubydebug { metadata => true } }
	}
}
10

You set "//var" instead of "/var" in your first grok pattern

11

My apologies. It was a typo, fixed now. However, it is not showing the correct type for logs in sub-directories. The grok works well for logs in the base directory.

12

It should work with examples you give.
But I guess you have different path names that you offuscate for confidential reasons I suppose.

To test/debug your grok patterns, I invite you to use :
https://grokdebug.herokuapp.com

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.