# Persist data between log lines with ruby filter **URL:** \ **Category:** Logstash\ **Created:** [October 31, 2019, 3:06pm UTC](https://discuss.elastic.co/t/persist-data-between-log-lines-with-ruby-filter/206068 "2019-10-31T15:06:13Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jong99](https://avatars.discourse-cdn.com/v4/letter/j/9dc877/32.png) [@jong99](https://discuss.elastic.co/u/jong99)\ **Post date:** [October 31, 2019, 3:06pm UTC](https://discuss.elastic.co/t/persist-data-between-log-lines-with-ruby-filter/206068/1 "2019-10-31T15:06:13Z")
What is the best way to persist data between two messages using the ruby filter? For example to include the previous message parsed I can use global variables like this: ``` input { generator { lines => [ "line 1", "line 2", "line 3" ] count => 1 } } filter { ruby { init => "$last_event = ''" code => " event.set('last_message', $last_event) unless $last_event.empty? $last_event = event.get('message') unless event.get('message').empty? " } } output { stdout { codec => rubydebug } } ``` Output: ``` { "sequence" => 0, "last_message" => "line 1", "message" => "line 2", "@version" => "1", "host" => "de9b70b8812b", "@timestamp" => 2019-10-31T14:36:11.669Z } { "sequence" => 0, "last_message" => "line 2", "message" => "line 3", "@version" => "1", "host" => "de9b70b8812b", "@timestamp" => 2019-10-31T14:36:11.669Z } { "@timestamp" => 2019-10-31T14:36:11.648Z, "@version" => "1", "sequence" => 0, "host" => "de9b70b8812b", "message" => "line 1" } ``` I understand the limitations of message ordering but was wondering what the recommended way of doing this was? I'd rather not use global variables unless absolutely necessary. Thanks! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 31, 2019, 5:15pm UTC](https://discuss.elastic.co/t/persist-data-between-log-lines-with-ruby-filter/206068/2 "2019-10-31T17:15:30Z")
> [@jong99](#): > > I'd rather not use global variables unless absolutely necessary. You should never need to use global variables. You can do this using an instance variable (@last\_event). You can only use a single worker thread, so the solution does not scale, and you have to use the ruby execution engine because with java\_execution enabled events get [re-ordered](https://github.com/elastic/logstash/issues/10938). ---
**Author:** ![jong99](https://avatars.discourse-cdn.com/v4/letter/j/9dc877/32.png) [@jong99](https://discuss.elastic.co/u/jong99)\ **Post date:** [November 1, 2019, 2:40pm UTC](https://discuss.elastic.co/t/persist-data-between-log-lines-with-ruby-filter/206068/3 "2019-11-01T14:40:37Z")
Thanks 🙂 I'll use instance variables. I'm not too concerned about the ordering, this is to get a fallback of an approximate time of a log line when the log line happens to be missing the timestamp. ---
**Author:** ![mogwaipr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mogwaipr/32/47584_2.png) [@mogwaipr](https://discuss.elastic.co/u/mogwaipr)\ **Post date:** [November 13, 2019, 7:02pm UTC](https://discuss.elastic.co/t/persist-data-between-log-lines-with-ruby-filter/206068/4 "2019-11-13T19:02:02Z")
I have a file that has a header, then a row with the data I need, then a header which I just drop then subsequent events I need to stick info from the row with the timestamp. this is not working for me. > Blockquote > if "" not in [CLLI] { > mutate { > add\_field =\> { > "timestamp" =\> "%{RPTDATE\_MONTH}/%{RPTDATE\_DAY}/%{RPTDATE\_YEAR} %{TIME}" > } > } > ruby { > init =\> '@@globals = Hash.new' > code =\> '@@globals["timestamp"] = event.get["timestamp"]; > @@globals["CLLI"] = event.get["CLLI"]; > @@globals["SWREL"] = event.get["SWREL"]; > @@globals["TZ"] = event.get["TZ"]; > @@globals["RTYPE"] = event.get["RTYPE"]; > @@globals["RPTPD"] = event.get["RPTPD"]; > @@globals["IVALDATE\_YEAR"] = event.get["IVALDATE\_YEAR"]; > @@globals["IVALDATE\_MONTH"] = event.get["IVALDATE\_MONTH"]; > @@globals["IVALDATE\_DAY"] = event.get["IVALDATE\_DAY"]; > @@globals["IVALSTART"] = event.get["IVALSTART"]; > @@globals["IVALEND"] = event.get["IVALEND"]; > @@globals["NUMENTIDS"] = event.get["NUMENTIDS"]; > ' > } > drop{} > }else{ > ruby{ > code =\> 'event.set("timestamp", @@globals["timestamp"]) > event.set("CLLI", @@globals["CLLI"]) > event.set("SWREL", @@globals["SWREL"]) > event.set("TZ", @@globals["TZ"]) > event.set("RTYPE", @@globals["RTYPE"]) > event.set("RPTPD", @@globals["RPTPD"]) > event.set("IVALDATE\_YEAR", @@globals["IVALDATE\_YEAR"]) > event.set("IVALDATE\_MONTH", @@globals["IVALDATE\_MONTH"]) > event.set("IVALDATE\_DAY", @@globals["IVALDATE\_DAY"]) > event.set("IVALSTART", @@globals["IVALSTART"]) > event.set("IVALEND", @@globals["IVALEND"]) > event.set("NUMENTIDS", @@globals["NUMENTIDS"]) > ' > } @last\_event becomes the last non-dropped event? I can't find documentation with @last\_event. should I just pull from @last\_event['fieldname'] ? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 13, 2019, 7:16pm UTC](https://discuss.elastic.co/t/persist-data-between-log-lines-with-ruby-filter/206068/5 "2019-11-13T19:16:19Z")
> [@mogwaipr](#): > > @last\_event becomes the last non-dropped event? The first post included this ruby code ``` $last_event = event.get('message') unless event.get('message').empty? ``` My point was that instead of a global $last\_event the poster should use an instance variable @last\_event. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 11, 2019, 7:44pm UTC](https://discuss.elastic.co/t/persist-data-between-log-lines-with-ruby-filter/206068/7 "2019-12-11T19:44:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.