Logstash conf multiline management

Tibo · May 26, 2015, 2:33pm

Hi,

I have a log file in input to logstash, something like that :
[....]
2015-04-08 00:21:21,248 [11 ] INFO Traitement de l'archive 20150313.tar-gz
2015-04-08 00:21:21,248 [11 ] INFO Extraction de l'archive 20150313.tar-gz
2015-04-08 00:21:57,513 [11 ] INFO Fin de l'extraction de l'archive
[....]

I already have an elapsed filter to calculate the processing time between the "Traitement" line and the "Fin de l'archive" one.
But I'd also like to index with the last line the name of the tar-gz file (here "20150313"). The issue is I need to get it from previous line.

Any help will be appreciated. Thanks

rafaltrojniak · May 26, 2015, 8:34pm

Hello Tibo,

Your problem was quite interesting, so doing some research work I used it.

The problem is : When the end event is processed, the beginning event processing is already finished. There is no direct way to access it.

The only idea on how to walk-around that from my POV is :

Simply add this information to final event on event generator side (application/script)
Use Multiline and connect both events into one.
The last way Proof Of Concept I had implemented on my githab tibo_case branch. If you would like to still have the separate events, you would have to duplicate the events.

Rules :

github.com

rafaltrojniak/logstash_rules/blob/tibo_case/rules/tibo.conf

filter {
  if [format] == "tibo_problem" {
    grok {
      match => [
        "message",
        "^%{TIMESTAMP_ISO8601:date} \[%{DATA:some_number}\] %{WORD:level} %{DATA:msg}$"
        ]
    }
    if [msg] =~ "Traitement de l'archive.*" {
      mutate{
        add_tag => [ "start_event" ]
      }
    }
    if [msg] == "Fin de l'extraction de l'archive" {
      mutate{
        add_tag => [ "end_event" ]
      }
    }
    elapsed {
      start_tag => "start_event"

This file has been truncated. show original

Doc :

github.com

rafaltrojniak/logstash_rules/blob/tibo_case/doc.md

Formats
-------

### Tibo issue

#### Event metadaa

#### Fields :
* `format`
  * Description : Just a format id
  * type: string
  * example values:
    * `"tibo_problem"`
* `msg`
  * Description : MSG part of entries
  * type: string
  * example values:
    * `"Traitement de l'archive 20150313.tar-gz\nExtraction de l'archive 20150313.tar-gz\nFin de l'extraction de l'archive"`
* `elapsed.time`
  * Description : Time generated by elapsed filter

This file has been truncated. show original

Hope that helps

Regards,

Tibo · May 27, 2015, 8:03am

Thanks a lot, exactly what I needed.

Regards