How to parse two different kind of format of log files at same time?

Elastic Stack Logstash
1

Hello,

I have 2 log files need to parse, which are aaa.log and bbb.log.
The contents of log files are different and no specific strings to differentiate both of them.
So I add type to differentiate them in input.
However, I have to launch two logstash to parse two different logs.
How do I use one logstash config to parse these two logs at same time?

My sample config is as below.

input {
    file {
        path => ["/log/*"]
        type => "aaa.log"
        # type => "bbb.log"
        start_position => "beginning"
    }
}

filter {
    if [type] == "aaa.log" {
        grok { match => {"message" => "%{AAA_FILE_NAME}"} }
    }
    else if [type] == "bbb.log" {
        grok{  match => {"message" => "%{BBB_FILE_NAME}"} }
    }
}

output { stdout { codec => rubydebug } }

--

Many thanks your reply.
Payton

2

You can specify multiple patterns, see the second example here - https://www.elastic.co/guide/en/logstash/2.1/plugins-filters-grok.html#plugins-filters-grok-match

3

Thanks your quick response.

Actually, we have over 10 different format type of log files.
Each kind of log file will use multiple patterns to parse our logs.
The major different of these log files are only file name.

We found a method to resolve this issue.
Use grok to get file name from first.

input {
    file {
        path => ["/test/*"]  #test1.log & test2.log in folder
        start_position => "beginning"
    }
}

filter {
    grok {
        match => {"path" => "%{GREEDYDATA}/%{GREEDYDATA:type}"}
    }
    if [type] == "test1.log" {
         grok {
            match => { ... }
        }
    }
    else if [type] == "test2.log" {
         grok {
            match => { ... }
        }
    }
}

--

Thanks a lot

4

This solution focuses on name of file. How do I parse different log files without knowing their name before ?
My log files can be named randomly by different network providers (I dont have control on how do they name log files.) I want to parse based on certain formats they have.

Do we know how to do that ?

Thanks,
Gaurav

5

you can use differnt grok matchers to parse each format