Persist data between log lines with ruby filter

[jong99]

What is the best way to persist data between two messages using the ruby filter? For example to include the previous message parsed I can use global variables like this:

input {
  generator {
    lines => [
      "line 1",
      "line 2",
      "line 3"
    ]
    count => 1
  }
}

filter {
  ruby {
    init => "$last_event = ''"
    code => "
      event.set('last_message', $last_event) unless $last_event.empty?
      $last_event = event.get('message') unless event.get('message').empty?
      "
  }
}

output { stdout { codec => rubydebug } }

Output:

{
        "sequence" => 0,
    "last_message" => "line 1",
         "message" => "line 2",
        "@version" => "1",
            "host" => "de9b70b8812b",
      "@timestamp" => 2019-10-31T14:36:11.669Z
}
{
        "sequence" => 0,
    "last_message" => "line 2",
         "message" => "line 3",
        "@version" => "1",
            "host" => "de9b70b8812b",
      "@timestamp" => 2019-10-31T14:36:11.669Z
}
{
    "@timestamp" => 2019-10-31T14:36:11.648Z,
      "@version" => "1",
      "sequence" => 0,
          "host" => "de9b70b8812b",
       "message" => "line 1"
}

I understand the limitations of message ordering but was wondering what the recommended way of doing this was? I'd rather not use global variables unless absolutely necessary.

Thanks!

[Badger]

@jong99

I'd rather not use global variables unless absolutely necessary.

You should never need to use global variables. You can do this using an instance variable (@last_event). You can only use a single worker thread, so the solution does not scale, and you have to use the ruby execution engine because with java_execution enabled events get re-ordered.

[jong99]

Thanks I'll use instance variables.

I'm not too concerned about the ordering, this is to get a fallback of an approximate time of a log line when the log line happens to be missing the timestamp.

[mogwaipr (Robert Lozano)]

I have a file that has a header, then a row with the data I need, then a header which I just drop then subsequent events I need to stick info from the row with the timestamp.

this is not working for me.

Blockquote
if "" not in [CLLI] {
mutate {
add_field => {
"timestamp" => "%{RPTDATE_MONTH}/%{RPTDATE_DAY}/%{RPTDATE_YEAR} %{TIME}"
}
}
ruby {
init => '@@globals = Hash.new'
code => '@@globals["timestamp"] = event.get["timestamp"];
@@globals["CLLI"] = event.get["CLLI"];
@@globals["SWREL"] = event.get["SWREL"];
@@globals["TZ"] = event.get["TZ"];
@@globals["RTYPE"] = event.get["RTYPE"];
@@globals["RPTPD"] = event.get["RPTPD"];
@@globals["IVALDATE_YEAR"] = event.get["IVALDATE_YEAR"];
@@globals["IVALDATE_MONTH"] = event.get["IVALDATE_MONTH"];
@@globals["IVALDATE_DAY"] = event.get["IVALDATE_DAY"];
@@globals["IVALSTART"] = event.get["IVALSTART"];
@@globals["IVALEND"] = event.get["IVALEND"];
@@globals["NUMENTIDS"] = event.get["NUMENTIDS"];
'
}
drop{}
}else{
ruby{
code => 'event.set("timestamp", @@globals["timestamp"])
event.set("CLLI", @@globals["CLLI"])
event.set("SWREL", @@globals["SWREL"])
event.set("TZ", @@globals["TZ"])
event.set("RTYPE", @@globals["RTYPE"])
event.set("RPTPD", @@globals["RPTPD"])
event.set("IVALDATE_YEAR", @@globals["IVALDATE_YEAR"])
event.set("IVALDATE_MONTH", @@globals["IVALDATE_MONTH"])
event.set("IVALDATE_DAY", @@globals["IVALDATE_DAY"])
event.set("IVALSTART", @@globals["IVALSTART"])
event.set("IVALEND", @@globals["IVALEND"])
event.set("NUMENTIDS", @@globals["NUMENTIDS"])
'
}

@last_event becomes the last non-dropped event? I can't find documentation with @last_event. should I just pull from @last_event['fieldname'] ?

[Badger]

@mogwaipr

@last_event becomes the last non-dropped event?

The first post included this ruby code

$last_event = event.get('message') unless event.get('message').empty?

My point was that instead of a global $last_event the poster should use an instance variable @last_event.