Multiline issue with log

Hi,

I'm working with a log with the following structure (fragment):

  2018-01-30 18:14:10.466554
  (0x03000000:PCDataField  ):folio                            = '201711231427280265' (CHARACTER)
  (0x03000000:PCDataField  ):idProceso                        = '7' (CHARACTER)
  (0x03000000:PCDataField  ):idSubproceso                     = '101' (CHARACTER)
  (0x03000000:PCDataField  ):idSubetapa                       = '273' (CHARACTER)
  (0x03000000:PCDataField  ):idSnapshot                       = '2064.c76a4df4-7e11-4c0a-9a04-f305faaef4d6' (CHARACTER)
  (0x01000000:Folder       ):parametros                       = (
    (0x01000000:Folder):parametro = (
      (0x03000000:PCDataField):nombre  = 'REPROCESO' (CHARACTER)
      (0x01000000:Folder     ):valores = (
        (0x03000000:PCDataField):valor = '2' (CHARACTER)
      )
    )
    (0x01000000:Folder):parametro = (
      (0x03000000:PCDataField):nombre  = 'ID_INSTANCIA' (CHARACTER)
      (0x01000000:Folder     ):valores = (
        (0x03000000:PCDataField):valor = '16649' (CHARACTER)
      )
    )....

each time a new event is generated a similar block is added to the log, I filter this new event using a codec multiline like this:

input {
file {
path => "/home/test.log"
start_position => "beginning"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => "previous"
}
}
}

Then when i try to get the fields (folio, idProceso, etc), using the following filter:

grok{
match => { "message" => "\W\w+:\w+\W:%{DATA:linea}\(" }
}

It only applies the filter to the first line of the multiline, for example it only returns: folio = '201711231427280265'

I don't know, how to change the filter to apply it again to the remaining lines of the log?

Thanks for your help

This category is for discussions in English. If you want to post in Spanish please use https://discuss.elastic.co/c/in-your-native-tongue/elastic-en-espanol.

Ok, thanks

I changed the post to English :slight_smile:

That grok filter is not valid (it has a trailing parenthesis). Can you fix that?

If I had to parse that I would go after it with something like this

    grok{
        break_on_match => false
        match => { "message" => [ 
                "\A%{DATA:ts}
",
                ":folio\s+= '%{DATA:folio}'",
                ":idProceso\s+= '%{DATA:idProceso}'",
                ":idSubproceso\s+= '%{DATA:idSubproceso}'",
                ":idSubetapa\s+= '%{DATA:idSubetapa}'",
                ":idSnapshot\s+= '%{DATA:idSnapshot}'"
            ]
        }
    }

Plus maybe this, although it is ugly and fragile.

    ruby {
        code => "
            m = event.get('message')
            m1 = m.scan( /nombre\s+= '(?<nombre>[^']+)'/ )
            m2 = m.scan( /valor = '(?<valor>[^']+)'/ )
            h = {}
            m1.to_a.each_index { |i|
                h[ m1[i][0].to_s ] = m2[i][0].to_s
            }
            event.set('parametros', h)
        "   
    }
1 Like

Thank you so much, it worked perfectly for my problem.

I have limited experience in ELK,it would be great if you could recommend some pages or books where I can learn more about ELK?

Looking forward to hearing from you.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.