# Replay old events as if they were new? **URL:** \ **Category:** Logstash\ **Created:** [March 24, 2021, 1:43pm UTC](https://discuss.elastic.co/t/replay-old-events-as-if-they-were-new/268218 "2021-03-24T13:43:41Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![GrahamHannington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grahamhannington/32/4404_2.png) [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)\ **Post date:** [March 24, 2021, 1:43pm UTC](https://discuss.elastic.co/t/replay-old-events-as-if-they-were-new/268218/1 "2021-03-24T13:43:42Z")
Suppose I have: - A Kibana dashboard that shows the last few minutes, automatically refreshing every few seconds - A Logstash config that reads JSON Lines from a file (via `stdin`), sets the event timestamp to the value of a particular field (in ISO 8601 combined date/time extended format: "_yyyy-mm-dd_T_hh:mm:ss.SSSSSSZ_"), and forwards it to Elasticsearch - A file of old JSON Lines events I want to "replay" that file of data as if it were new; I want a Logstash config that shifts past timestamps to the present. That is, I want a Logstash config that: 1. Updates the timestamp in the first line to the current time 2. Update the timestamps in subsequent lines _by the same amount_ This seems like a reasonably common use case, but I haven't found an existing filter that does this. Before I reinvent this wheel, I thought I'd ask here. My existing Logstash config: ```auto input { stdin { codec => json_lines } } filter { date { match => ["time", "ISO8601"] } } output { elasticsearch { hosts => ["http://elastic.my.com:9200"] index => "mydata-%{+YYYY.MM.dd}" manage_template => false } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 24, 2021, 3:18pm UTC](https://discuss.elastic.co/t/replay-old-events-as-if-they-were-new/268218/2 "2021-03-24T15:18:15Z")
I think it is a rather unusual use case, but I think this does it: ``` ruby { code => ' if ! @offset @offset = Time.now.to_f - event.get("@timestamp").to_f end event.set("@timestamp", LogStash::Timestamp.new(Time.at(event.get("@timestamp").to_f + @offset))) ' } ``` Set pipeline.workers to 1 and ensure pipeline.ordered has the right value. ---
**Author:** ![GrahamHannington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grahamhannington/32/4404_2.png) [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)\ **Post date:** [March 25, 2021, 11:54am UTC](https://discuss.elastic.co/t/replay-old-events-as-if-they-were-new/268218/3 "2021-03-25T11:54:16Z")
That works, thank you! However, it triggers the following disconcerting warnings: > WARNING: An illegal reflective access operation has occurred > WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/C:/tools/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby\_stdin\_channel/jruby\_stdin\_channel.jar) to field java.io.FilterInputStream.in > WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader > WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations > WARNING: All illegal access operations will be denied in a future release I look at those warnings and see the deep, dark rabbit hole of my own ignorance. Can you tweak your Ruby code to avoid the, um, "illegal reflective access operation"? Or is this a [known issue](https://github.com/jruby/jruby/issues/4834) that I can safely ignore? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 25, 2021, 6:39pm UTC](https://discuss.elastic.co/t/replay-old-events-as-if-they-were-new/268218/4 "2021-03-25T18:39:31Z")
It is a [known issue](https://github.com/colinsurprenant/jruby-stdin-channel/issues/1). You can ignore it. ---
**Author:** ![GrahamHannington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grahamhannington/32/4404_2.png) [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)\ **Post date:** [March 26, 2021, 5:08am UTC](https://discuss.elastic.co/t/replay-old-events-as-if-they-were-new/268218/5 "2021-03-26T05:08:40Z")
Thanks again, Badger. In the future, I can see myself needing to time-shift other time stamp fields in each event (not just `@timestamp`; not just the "primary" time stamp field for the event). I'm not yet clear on how to do that. Currently, I think that: - `date` creates `@timestamp` as a Ruby `Time` object based on the string value of (in my case) the `time` field - `Time.now` also returns a `Time` object - We apply the `.to_f` method to both of these `Time` objects, and we get the offset between them I'm not sure of the most elegant, most efficient way to do that for other fields in the JSON Lines whose values happen to be time stamps (as opposed to the field that is to be used as the time stamp for the event, the index, as a whole). I'll just park that thought here for now. If it's easy, then I'd be grateful for tips here. Otherwise, when I hit that issue for real, I'll play around with some alternatives myself, and create a new topic. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 26, 2021, 4:58pm UTC](https://discuss.elastic.co/t/replay-old-events-as-if-they-were-new/268218/6 "2021-03-26T16:58:01Z")
If you need to parse additional time fields use a date filter and set the target option to save the result in a different field. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 23, 2021, 4:58pm UTC](https://discuss.elastic.co/t/replay-old-events-as-if-they-were-new/268218/7 "2021-04-23T16:58:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.