Date filter plugin issue

[dyl (bus)]

Hey,

I try to parse a duration field thanks to "date" filter plugin but i'm unable to do this correctly ><

My log : 0h 41' 47" 175ms

I tried :

H'h' MM\' ss'"' SSS'ms'

But it doesn't work..I am a little confused with the ' and " etc...

Can somebody help me to find the good date filter ?

Some exemple of my logs :

0h 1' 24" 811ms
0h 1' 51" 430ms
0h 2' 0" 73ms
0h 41' 2" 493ms
.
.

Thx u !

[magnusbaeck (Magnus Bäck)]

The date filter doesn't parse durations.

[dyl (bus)]

Hm ok, so how can I do to handle with duration field ?

I want to do that to do a query which search all event where this field is > 2 minutes for exemple. Is it possible to do this kind of "where" query ?

[magnusbaeck (Magnus Bäck)]

The easiest option is probably to write some Ruby code in a ruby filter.

[dyl (bus)]

I don't know ruby :s

Do you have any ideas to help me?

[magnusbaeck (Magnus Bäck)]

Sorry, I don't have time to give detailed guidance.

[dyl (bus)]

Maybe @Badger can help me, will see..

But thx for help !

[Badger]

You could parse it using grok. If you really do just want to test for the number of minutes being greater than two then you don't even need to combine the pieces.

grok { match => [ "message", "%{NUMBER:h:int}h %{NUMBER:m:int}' %{NUMBER:s:int}\" %{NUMBER:ms:int}ms" ] }

[dyl (bus)]

@Badger

%{NUMBER:h:int}h %{NUMBER:m:int}' %{NUMBER:s:int}" %{NUMBER:ms:int}ms

Ok and with this grok i would like to do : all my events where "m" > "2". Can I do that ?

[magnusbaeck (Magnus Bäck)]

if [m] > 2 {
    ...
}

Now of course this won't catch a duration like "1h 0m 0s" which is why I suggested a Ruby-based solution that produces a single integer representing the total number of milliseconds (or whatever unit of resolution is desired).

[dyl (bus)]

So ok I can do that into my logstash pipeline,

but me I would like to that into Kibana, for exemple to have the name of my event where m > 2

[magnusbaeck (Magnus Bäck)]

The query string duration:[2000000 TO *] finds all events where the duration field is 2000000 or greater.

[dyl (bus)]

Ok thx so I have 3 format of duration log :

0h 1' 1" 646ms
2 mn 35 s 288 ms
24 s 515 ms

I did my grok, it work well :

 grok
    {
      match => { "DESCRIPTION" => ["%{NUMBER:HEURE:int}h %{NUMBER:MINUTE:int}' %{NUMBER:SECONDE:int}\" %{NUMBER:MILLISECONDE:int}ms","%{NUMBER:MINUTE:int} mn %{NUMBER:SECONDE:int} s %{NUMBER:MILLISECONDE:int} ms","%{NUMBER:SECONDE:int} s %{NUMBER:MILLISECONDE:int} ms"] }
    }

But I save all the fields : (for further use)

ruby { code => "@@save_the_heure = event.get('HEURE')" }
ruby { code => "@@save_the_minute = event.get('MINUTE')" }
ruby { code => "@@save_the_seconde = event.get('SECONDE')" }
ruby { code => "@@save_the_milliseconde = event.get('MILLISECONDE')" }

And sometime (for exemple my second log line exemple), my field "HEURE" is empty.. logic, so I want to add 0 instead of nil and then to convert in integer but it doesn't work like that :

if !([HEURE]) 
{
  mutate
  {
    add_field => { "HEURE" => "0" }
    convert => { "HEURE" => "integer" }
  }	
}

Have you got another solution ?

EDIT : Solution :

if !([HEURE]) 
{
  mutate { add_field => { "HEURE" => 0 } }
  mutate { convert => ["HEURE","integer"] }
}