Multiple timestamped log file format

Elastic Stack Logstash
1

Hi, i have two timestamped log file names "server_log11032019", "app_log11032019" and in one of those log file there is multiple format for example(app_log) :
2019-03-11 10:53:55,741 ERROR [com.website.classname] Error_msg Java_exception
2019-03-11 10:53:55,741 ERROR [com.website.classname] ORACLE_DB_ERR
.
.
i think the filter will be like this :
filter {
if [type] == "server_log" {
grok { match => {"message" => "%{AAA_FILE_NAME}"} }
}
else if [type] == "app_log" {
grok{
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} [(?[^]]+)] %{WORD:Message d'erreur}(?<exception'>[^]]+)"}
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} [(?<classnamee'>[^]]+)] (?<mssg'>[^]]+)"}
}
}
}
can we do that ?

2

Yes, you can have multiple matches inside of a grok.

You could handle this using an or clause in your regex (|)

grok
{
  match =&gt; {"message" =&gt; "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel}( [(?[^]]+)] %{WORD:Message d'erreur}(?&lt;exception'&gt;[^]]+))|([(?&lt;classnamee'&gt;[^]]+)] (?&lt;mssg'&gt;[^]]+))"}
}

You can use the | as an or clause in your regex.

Another way to handle it would be to do an if else if

if "Error_msgJava_Exception" in [message] {
  grok {
    match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} [(?[^]]+)] %{WORD:Message d'erreur}(?<exception'>[^]]+)"}
  }
}
else if "ORACLE_DB_ERR" in [message] {
  grok {
    match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} [(?<classnamee'>[^]]+)] (?<mssg'>[^]]+)"}
  }
}

Are you running into an error or are you just trying to come up with the best way of handling it?
If you want suggestions for handling multiple scenarios, then we would need a longer log example to play with so that we can make better suggestions. In my case I use the "if else if" a lot for breaking up my logs. Usually I only have a handful of unique log types in a single file that I need to process.

3

what about the files names "app_log","server_log", because i have the two files in the same folder how i can write the condition ?
if [path/source/type] what keyword we use for that ?
thank you.

4

Have you checked your output from processing a log?

You should have something like ["log"]["file"]["path"] that you can use to pull the name of your log file and do an if else if on that.

if {
  "server_log" in ["log"]["file"]["path"] {
    do stuff
  }
}

Please note I did not test the above code, this is just off the top of my head to point you in the correct direction.

5

the first solution is not suitable for my case, and for the second one i have a log file that contains multiple log format and i can't make "if else if" for all of them any other suggestions !!

6

Can you give some more examples then? I'm not sure what you are trying to do.

If you want to process different files in different ways then you will have to have a conditional somewhere. That or you have to a different port for every log file that you process.
You can also do pipeline to pipeline processing. That way your conditionals sit either in your pipelines.yml or into a routing pipeline.

7

it's working using this :

if [source] =~ "server_log" {
         grok {
            match => {"message" => "%{TIMESTAMP_ISO8601:Timestamp}\|(\[%{BASE10$
        }
    }
 else if [source] =~ "app_log" {

        grok {
             match =>{ "message" => ["%{TIMESTAMP_ISO8601:Timestamp}.......","%{TIMESTAMP_ISO8601:Timestamp}.......", ...]}
    }  
 }
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.