Parse with different grok filter problem

Elastic Stack Logstash
1

Hello everybody, I want to parse these events :

INFO;0000;000003;data/EDT/batchs/files/logs/REL015MT-20180807-20180807-2004.log

INFO;0000;000003; /data/EDT/batchs/files/logs/WKF997-20180904-20180904-2029-42.log
INFO;0000;000003; /data/EDT/batchs/file/logs/WKF997-20180611-20180611-2035-42-9000-01-20180611-203557.log
INFO;0000;000003; /data/EDT/batchs/files/logs/WKF999-20180904-20180904-2003-40.log
INFO;0000;000003; /data/EDT/batchs/file/logs/WKF999-20180904-20180904-1958-20-42-2297-1168-0-1705969074-20180904-200117.log

INFO;0000;000003; /data/EDT/batchs/files/logs/E-0077MT-20180810-20180810-2317.log

And to parse these logs I use 3 filter grok different, in same order than above :

%{WORD:NOM_BATCH}-%{BASE16NUM:DATE_JOURNEE_BATCH}-%{BASE16NUM:DATE_EXECUTION_BATCH}-%{BASE16NUM:HEURE_EXECUTION_BATCH}.log

%{WORD:NOM_BATCH}-%{BASE16NUM:DATE_JOURNEE_BATCH}-%{BASE16NUM:DATE_EXECUTION_BATCH}-%{BASE16NUM:HEURE_EXECUTION_BATCH}-%{GREEDYDATA:SUITE}.log

(?<NOM_BATCH>E-[0-9A-Z]+)-%{BASE16NUM:DATE_JOURNEE_BATCH}-%{BASE16NUM:DATE_EXECUTION_BATCH}-%{BASE16NUM:HEURE_EXECUTION_BATCH}.log

So all these grok works fine but I want to group all these filter to filter all event.

For exemple if the first is grok parse failure, pass to the next, etc ...

I tried this :

filter
{
  grok {
    match => { "message" => [ "%{WORD:TYPE};%{DATA:ID1};%{NUMBER:ID2};%{GREEDYDATA:DESCRIPTION}" ] }
  }
  
  if ([DESCRIPTION] =~ "\/data\/EDT\/batchs\/files\/logs\/[A-Za-z0-9_.]+.+.log")
  {
    grok {
      match => { "DESCRIPTION" => [ "%{WORD:NOM_BATCH}-%{BASE16NUM:DATE_JOURNEE_BATCH}-%{BASE16NUM:DATE_EXECUTION_BATCH}-%{BASE16NUM:HEURE_EXECUTION_BATCH}.log" ] }
      add_field => { "PATH_BATCH" => "%{NOM_BATCH}-%{DATE_JOURNEE_BATCH}-%{DATE_EXECUTION_BATCH}-%{HEURE_EXECUTION_BATCH}.log" }
    }
    grok {
      match => { "DESCRIPTION" => [ "%{WORD:NOM_BATCH}-%{BASE16NUM:DATE_JOURNEE_BATCH}-%{BASE16NUM:DATE_EXECUTION_BATCH}-%{BASE16NUM:HEURE_EXECUTION_BATCH}-%{GREEDYDATA:SUITE}.log" ] }
      add_field => { "PATH_BATCH" => "%{NOM_BATCH}-%{DATE_JOURNEE_BATCH}-%{DATE_EXECUTION_BATCH}-%{HEURE_EXECUTION_BATCH}-%{SUITE}.log" }
    }
    grok {
      match => { "DESCRIPTION" => [ "(?<NOM_BATCH>E-[0-9A-Z]+)-%{BASE16NUM:DATE_JOURNEE_BATCH}-%{BASE16NUM:DATE_EXECUTION_BATCH}-%{BASE16NUM:HEURE_EXECUTION_BATCH}.log" ] }
      add_field => { "PATH_BATCH" => "%{NOM_BATCH}-%{DATE_JOURNEE_BATCH}-%{DATE_EXECUTION_BATCH}-%{HEURE_EXECUTION_BATCH}.log" }
    }
  }
}

but it doesn't work as expected..

Can somebody help me ? ty

3

Thx u for answer but I need to add a field specific to a grok filter so I can't group all these filter into 1 pattern

4

Parsing is not done properly I guess, please check out the following conf

input {
    file {
        path => ["~/data.log"]
        start_position => "beginning"
    }
}

filter {
    grok {
        match => { "message" => "%{WORD:TYPE};%{DATA:ID1};%{NUMBER:ID2};%{GREEDYDATA:DESCRIPTION}" }
    }

    if ([DESCRIPTION] =~ "data\/EDT\/batchs\/files\/logs\/[A-Za-z0-9_.]+.+.log")
    {
        grok {
            match => { "DESCRIPTION" => "^data\/EDT\/batchs\/files\/logs\/%{WORD:NOM_BATCH}-%{BASE16NUM:DATE_JOURNEE_BATCH}-%{BASE16NUM:DATE_EXECUTION_BATCH}-%{BASE16NUM:HEURE_EXECUTION_BATCH}.log$" }
            add_field => { "PATH_BATCH" => "1" }
        }
        if (![PATH_BATCH]) {
            grok {
                match => { "DESCRIPTION" => "^ \/data\/EDT\/batchs\/files\/logs\/%{WORD:NOM_BATCH}-%{BASE16NUM:DATE_JOURNEE_BATCH}-%{BASE16NUM:DATE_EXECUTION_BATCH}-%{BASE16NUM:HEURE_EXECUTION_BATCH}-%{GREEDYDATA:SUITE}.log$" }
                add_field => { "PATH_BATCH" => "2" }
            }
        }
        if (![PATH_BATCH]) {
            grok {
                match => { "DESCRIPTION" => "^ \/data\/EDT\/batchs\/files\/logs\/(?<NOM_BATCH>E-[0-9A-Z]+)-%{BASE16NUM:DATE_JOURNEE_BATCH}-%{BASE16NUM:DATE_EXECUTION_BATCH}-%{BASE16NUM:HEURE_EXECUTION_BATCH}.log$" }
                add_field => { "PATH_BATCH" => "3" }
            }
        }
    }
}

output {
    stdout { codec => rubydebug }
}

I was not able to find out why GREEDYDATA is not reading the super long stuff or if it's failing in if condition itself for that WKF<super long>.log lines.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.