Text based log format, need help in parsing

Elastic Stack Logstash
1

Hey All,

I have a log format which looks like this.

I need to extract all the key value based pairs. My current configuration looks like:

filter{
if [message] =~ /^#/ {
  drop {}
} else if [message] =~ /^\s*$/ {
  drop {}
} else if [message] =~ /^[A-Za-z ]+:/ {
  mutate { strip => [ "message" ] }
  ruby {
     init => '
            @@data = {}
        '
        code => '
            msg = event.get("message")
            matches = msg.scan(/^([A-Za-z]+):(.*)/)
            m = matches[0]
            @@data[m[0]] = m[1]
        '
    }
   drop{}
} else if [message] =~ /^[ 0-9]+/ {
    mutate { strip => [ "message" ] }
    grok { match => { "message" => '^[0-9]{1,}[.][ +](?<key>[^:]+):\s*%{GREEDYDATA:value}' } }
    ruby {
        code => '
            @@data[key] = value
        '
    }
    drop{}
} else if [message] =~ /^Speed/ {
    ruby{
        code => '
            event.set("data", @@data)
        '
    }
} else {
    drop{}
}

}

This results in a ruby exception [2019-05-22T12:11:00,169][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method' for nil:NilClass
`.

Any ideas or better solutions?

2

That will occur if you make an array index reference to a variable that is not an array.

3 
msg = event.get("message")
        matches = msg.scan(/^([A-Za-z]+):(.*)/)
        m = matches[0]
        @@data[m[0]] = m[1]

The above code seems to be working fine when I run it separately. Also I removed

else if [message] =~ /^[ 0-9]+/ {
mutate { strip => [ "message" ] }
grok { match => { "message" => '^[0-9]{1,}[.][ +](?<key>[^:]+):\s*%{GREEDYDATA:value}' } }
ruby {
    code => '
        @@data[key] = value
    '
}
drop{}

}
from the filter, still I get the same issue.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.