Persist data between log lines with ruby filter

What is the best way to persist data between two messages using the ruby filter? For example to include the previous message parsed I can use global variables like this:

input {
  generator {
    lines => [
      "line 1",
      "line 2",
      "line 3"
    ]
    count => 1
  }
}

filter {
  ruby {
    init => "$last_event = ''"
    code => "
      event.set('last_message', $last_event) unless $last_event.empty?
      $last_event = event.get('message') unless event.get('message').empty?
      "
  }
}

output { stdout { codec => rubydebug } }

Output:

{
        "sequence" => 0,
    "last_message" => "line 1",
         "message" => "line 2",
        "@version" => "1",
            "host" => "de9b70b8812b",
      "@timestamp" => 2019-10-31T14:36:11.669Z
}
{
        "sequence" => 0,
    "last_message" => "line 2",
         "message" => "line 3",
        "@version" => "1",
            "host" => "de9b70b8812b",
      "@timestamp" => 2019-10-31T14:36:11.669Z
}
{
    "@timestamp" => 2019-10-31T14:36:11.648Z,
      "@version" => "1",
      "sequence" => 0,
          "host" => "de9b70b8812b",
       "message" => "line 1"
}

I understand the limitations of message ordering but was wondering what the recommended way of doing this was? I'd rather not use global variables unless absolutely necessary.

Thanks!

You should never need to use global variables. You can do this using an instance variable (@last_event). You can only use a single worker thread, so the solution does not scale, and you have to use the ruby execution engine because with java_execution enabled events get re-ordered.

Thanks :slight_smile: I'll use instance variables.

I'm not too concerned about the ordering, this is to get a fallback of an approximate time of a log line when the log line happens to be missing the timestamp.

I have a file that has a header, then a row with the data I need, then a header which I just drop then subsequent events I need to stick info from the row with the timestamp.

this is not working for me.

Blockquote
if "" not in [CLLI] {
mutate {
add_field => {
"timestamp" => "%{RPTDATE_MONTH}/%{RPTDATE_DAY}/%{RPTDATE_YEAR} %{TIME}"
}
}
ruby {
init => '@@globals = Hash.new'
code => '@@globals["timestamp"] = event.get["timestamp"];
@@globals["CLLI"] = event.get["CLLI"];
@@globals["SWREL"] = event.get["SWREL"];
@@globals["TZ"] = event.get["TZ"];
@@globals["RTYPE"] = event.get["RTYPE"];
@@globals["RPTPD"] = event.get["RPTPD"];
@@globals["IVALDATE_YEAR"] = event.get["IVALDATE_YEAR"];
@@globals["IVALDATE_MONTH"] = event.get["IVALDATE_MONTH"];
@@globals["IVALDATE_DAY"] = event.get["IVALDATE_DAY"];
@@globals["IVALSTART"] = event.get["IVALSTART"];
@@globals["IVALEND"] = event.get["IVALEND"];
@@globals["NUMENTIDS"] = event.get["NUMENTIDS"];
'
}
drop{}
}else{
ruby{
code => 'event.set("timestamp", @@globals["timestamp"])
event.set("CLLI", @@globals["CLLI"])
event.set("SWREL", @@globals["SWREL"])
event.set("TZ", @@globals["TZ"])
event.set("RTYPE", @@globals["RTYPE"])
event.set("RPTPD", @@globals["RPTPD"])
event.set("IVALDATE_YEAR", @@globals["IVALDATE_YEAR"])
event.set("IVALDATE_MONTH", @@globals["IVALDATE_MONTH"])
event.set("IVALDATE_DAY", @@globals["IVALDATE_DAY"])
event.set("IVALSTART", @@globals["IVALSTART"])
event.set("IVALEND", @@globals["IVALEND"])
event.set("NUMENTIDS", @@globals["NUMENTIDS"])
'
}

@last_event becomes the last non-dropped event? I can't find documentation with @last_event. should I just pull from @last_event['fieldname'] ?

The first post included this ruby code

$last_event = event.get('message') unless event.get('message').empty?

My point was that instead of a global $last_event the poster should use an instance variable @last_event.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.