Multiline issue with log

[AGarcMan]

Hi,

I'm working with a log with the following structure (fragment):

  2018-01-30 18:14:10.466554
  (0x03000000:PCDataField  ):folio                            = '201711231427280265' (CHARACTER)
  (0x03000000:PCDataField  ):idProceso                        = '7' (CHARACTER)
  (0x03000000:PCDataField  ):idSubproceso                     = '101' (CHARACTER)
  (0x03000000:PCDataField  ):idSubetapa                       = '273' (CHARACTER)
  (0x03000000:PCDataField  ):idSnapshot                       = '2064.c76a4df4-7e11-4c0a-9a04-f305faaef4d6' (CHARACTER)
  (0x01000000:Folder       ):parametros                       = (
    (0x01000000:Folder):parametro = (
      (0x03000000:PCDataField):nombre  = 'REPROCESO' (CHARACTER)
      (0x01000000:Folder     ):valores = (
        (0x03000000:PCDataField):valor = '2' (CHARACTER)
      )
    )
    (0x01000000:Folder):parametro = (
      (0x03000000:PCDataField):nombre  = 'ID_INSTANCIA' (CHARACTER)
      (0x01000000:Folder     ):valores = (
        (0x03000000:PCDataField):valor = '16649' (CHARACTER)
      )
    )....

each time a new event is generated a similar block is added to the log, I filter this new event using a codec multiline like this:

input {
file {
path => "/home/test.log"
start_position => "beginning"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => "previous"
}
}
}

Then when i try to get the fields (folio, idProceso, etc), using the following filter:

grok{
match => { "message" => "\W\w+:\w+\W:%{DATA:linea}\(" }
}

It only applies the filter to the first line of the multiline, for example it only returns: folio = '201711231427280265'

I don't know, how to change the filter to apply it again to the remaining lines of the log?

Thanks for your help

[magnusbaeck (Magnus Bäck)]

This category is for discussions in English. If you want to post in Spanish please use https://discuss.elastic.co/c/in-your-native-tongue/elastic-en-espanol.

[AGarcMan]

Ok, thanks

[AGarcMan]

I changed the post to English

[Badger]

That grok filter is not valid (it has a trailing parenthesis). Can you fix that?

If I had to parse that I would go after it with something like this

    grok{
        break_on_match => false
        match => { "message" => [ 
                "\A%{DATA:ts}
",
                ":folio\s+= '%{DATA:folio}'",
                ":idProceso\s+= '%{DATA:idProceso}'",
                ":idSubproceso\s+= '%{DATA:idSubproceso}'",
                ":idSubetapa\s+= '%{DATA:idSubetapa}'",
                ":idSnapshot\s+= '%{DATA:idSnapshot}'"
            ]
        }
    }

Plus maybe this, although it is ugly and fragile.

    ruby {
        code => "
            m = event.get('message')
            m1 = m.scan( /nombre\s+= '(?<nombre>[^']+)'/ )
            m2 = m.scan( /valor = '(?<valor>[^']+)'/ )
            h = {}
            m1.to_a.each_index { |i|
                h[ m1[i][0].to_s ] = m2[i][0].to_s
            }
            event.set('parametros', h)
        "   
    }

[AGarcMan]

Thank you so much, it worked perfectly for my problem.

I have limited experience in ELK,it would be great if you could recommend some pages or books where I can learn more about ELK?

Looking forward to hearing from you.