Replay old events as if they were new?

[GrahamHannington]

Suppose I have:

• A Kibana dashboard that shows the last few minutes, automatically refreshing every few seconds
• A Logstash config that reads JSON Lines from a file (via stdin), sets the event timestamp to the value of a particular field (in ISO 8601 combined date/time extended format: "yyyy-mm-ddThh:mm:ss.SSSSSSZ"), and forwards it to Elasticsearch
• A file of old JSON Lines events

I want to "replay" that file of data as if it were new; I want a Logstash config that shifts past timestamps to the present.

That is, I want a Logstash config that:

1. Updates the timestamp in the first line to the current time
2. Update the timestamps in subsequent lines by the same amount

This seems like a reasonably common use case, but I haven't found an existing filter that does this. Before I reinvent this wheel, I thought I'd ask here.

My existing Logstash config:

input {
  stdin {
    codec => json_lines
  }
}
filter {
  date {
    match => ["time", "ISO8601"]
  }
}
output {
  elasticsearch {
    hosts => ["http://elastic.my.com:9200"]
    index => "mydata-%{+YYYY.MM.dd}"
    manage_template => false
  }
}

[Badger]

I think it is a rather unusual use case, but I think this does it:

    ruby {
        code => '
            if ! @offset
                @offset = Time.now.to_f - event.get("@timestamp").to_f
            end
            event.set("@timestamp", LogStash::Timestamp.new(Time.at(event.get("@timestamp").to_f + @offset)))
        '
    }

Set pipeline.workers to 1 and ensure pipeline.ordered has the right value.

[GrahamHannington]

That works, thank you!

However, it triggers the following disconcerting warnings:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/C:/tools/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in
WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I look at those warnings and see the deep, dark rabbit hole of my own ignorance.

Can you tweak your Ruby code to avoid the, um, "illegal reflective access operation"?

Or is this a known issue that I can safely ignore?

[Badger]

It is a known issue. You can ignore it.

[GrahamHannington]

Thanks again, Badger.

In the future, I can see myself needing to time-shift other time stamp fields in each event (not just @timestamp; not just the "primary" time stamp field for the event). I'm not yet clear on how to do that. Currently, I think that:

• date creates @timestamp as a Ruby Time object based on the string value of (in my case) the time field
• Time.now also returns a Time object
• We apply the .to_f method to both of these Time objects, and we get the offset between them

I'm not sure of the most elegant, most efficient way to do that for other fields in the JSON Lines whose values happen to be time stamps (as opposed to the field that is to be used as the time stamp for the event, the index, as a whole).

I'll just park that thought here for now. If it's easy, then I'd be grateful for tips here. Otherwise, when I hit that issue for real, I'll play around with some alternatives myself, and create a new topic.

[Badger]

If you need to parse additional time fields use a date filter and set the target option to save the result in a different field.