```
Data is coming from a SQL database. Is this even possible? I'm trying to do it in a Ruby filter using LogStash::Event.new(), but I'm clearly missing something (including complete documentation) since I end up with all kinds of filewatch errors and warnings when I do this. Another, earlier Ruby filter called in the .conf calculates the number of slices the event will require. Now I'm looking to actually generate five new events from the first one. All in the pursuit of being able to display historical data of resources used.
Any explanation here or pointers to how LogStash::Event.new() actually works would be fine.
Here's how I'm currently _trying_ to do this based upon the few threads I've been able to find that are kind of similar:
```auto
#!/usr/bin/env ruby
# the value of `params` is the value of the hash passed to `script_params`
# in the logstash configuration
def register(params)
#@times = params["time_field"]
end
# the filter method receives an event and must return a list of events.
# Dropping an event means not including it in the return array,
# while creating new ones only requires you to add a new instance of
# LogStash::Event to the returned array
def filter(event)
require 'time'
# The number of necessary slices was found in a previous
# filter, so generate that number of events with each
# event having its own timestamp that is set 15 minutes
# later than the previous event.
num_of_slices = event.get( "num_of_slices" )
running_timestamp = Time.parse( event.get( "@opentimestamp" ) )
end_timestamp = Time.parse( event.get( "@closetimestamp" ) )
runind = event.get( "runidx" )
num_of_slices.to_i.times { |i|
newhash = event.to_hash
gen = LogStash::Event.new( newhash )
gen.set( "runidxseq", runind.to_s + "_" + i.to_s )
gen.set( "@slice_timestamp", running_timestamp )
gen.set( "@timestamp_common", running_timestamp )
if num_of_slices > 1
if running_timestamp + 900 > end_timestamp
gen.set( "@slice_timestamp", end_timestamp )
gen.set( "@timestamp_common", end_timestamp )
end
running_timestamp = running_timestamp + 900
end
new_event_block.call( gen )
}
# First step recreated this event, so drop the incoming version.
event.cancel
end
```
@timestamp\_common is the agreed 'reference' timestamp field to allow multiple indices to plot together with a single proper x-axis date histogram. Doing it here rather than in the .conf file as a mutate operation.
Thank you for any help.
nick
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 6, 2024, 9:11pm UTC](https://discuss.elastic.co/t/splitting-records-with-start-and-end-times-into-separate-documents-with-a-timestamp-every-x-minutes/360997/2 "2024-06-06T21:11:24Z")
Another way of doing it would be
```
input { generator { count => 1 lines => [
'2024-06-06T16:37:33.463Z,2024-06-06T17:10:00.123Z,Foo',
'2024-06-06T16:45:00.250Z,2024-06-06T17:30:00.500Z,Bar',
'2024-06-06T17:00:00.000Z,2024-06-06T17:15:00.000Z,Baz'
] } }
filter {
csv { columns => ["startTime", "endTime", "label"] }
ruby {
code => '
startTime = Time.parse( event.get("startTime") )
endTime = Time.parse( event.get("endTime") )
numSlices = ( (endTime - startTime) / 900 ).ceil
event.set("@timestamp", LogStash::Timestamp.new(startTime))
currentTime = startTime + 900
while currentTime < endTime
clone = event.clone
clone.set("@timestamp", LogStash::Timestamp.new(currentTime))
new_event_block.call(clone)
currentTime += 900
end
clone = event.clone
clone.set("@timestamp", LogStash::Timestamp.new(endTime))
new_event_block.call(clone)
'
}
```
---
### Author:  [@nickbarry](https://discuss.elastic.co/u/nickbarry)
#### Post date: [June 7, 2024, 3:35am UTC](https://discuss.elastic.co/t/splitting-records-with-start-and-end-times-into-separate-documents-with-a-timestamp-every-x-minutes/360997/3 "2024-06-07T03:35:06Z")
That... looks brilliant and like it might work with some tweaks for my situation. I will give it a try, and if successful, I will respond back here so others can also benefit.
Thank you!
nick
---
### Author:  [@nickbarry](https://discuss.elastic.co/u/nickbarry)
#### Post date: [July 16, 2024, 3:36pm UTC](https://discuss.elastic.co/t/splitting-records-with-start-and-end-times-into-separate-documents-with-a-timestamp-every-x-minutes/360997/4 "2024-07-16T15:36:18Z")
Ok. I've finally got the time in my schedule to come back to this. It seems like it will work fine but I have one question: If my @opentimestamp and @closetimestamp are already LogStash::Timestamp objects by the time they get here, where can I find documentation on how to convert them to proper Ruby Time objects? I've tried just simple
```auto
open_time = event.get( "@opentimestamp" )
```
I've tried
```auto
open_time = Time.new( event.get( "@opentimestamp" ) )
```
And also
```auto
open_time = Time.parse( event.get( "@opentimestamp" ) )
```
None of these is correct. But without having them as Ruby Time objects, the
```auto
while curr_time < close_time
```
conditional always fails.
It's entirely possible that my search-fu is out of practice (I've tried DDG and Kagi, Google is just going to be garbage SEO results) but I can't find how to convert LogStash::Timestamp to a Ruby Time object IN THE RUBY SCRIPT. I understand I'll then have to cast back to LogStash::Timestamp on clone.set().
Once I have a working filter, I will post it in its entirety here in this thread since I can't be the only one who is doing this.
Thanks!
nick
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 16, 2024, 6:21pm UTC](https://discuss.elastic.co/t/splitting-records-with-start-and-end-times-into-separate-documents-with-a-timestamp-every-x-minutes/360997/5 "2024-07-16T18:21:52Z")
> [@nickbarry](#):
>
> I can't find how to convert LogStash::Timestamp to a Ruby Time object IN THE RUBY SCRIPT.
Use Time.at. Example [here](https://discuss.elastic.co/t/changing-time-format/278115/2).
---
### Author:  [@nickbarry](https://discuss.elastic.co/u/nickbarry)
#### Post date: [July 18, 2024, 3:17pm UTC](https://discuss.elastic.co/t/splitting-records-with-start-and-end-times-into-separate-documents-with-a-timestamp-every-x-minutes/360997/6 "2024-07-18T15:17:28Z")
This code works!
```auto
ruby {
code => '
require "time"
open_time = Time.at( event.get( "@opentimestamp" ).to_f )
close_time = Time.at( event.get( "@closetimestamp" ).to_f )
runidx = event.get( "runidx" )
i = 0
runidxseq = runidx.to_s + "_" + i.to_s
# First slice.
event.set( "runidxseq", runidxseq )
event.set( "@timestamp_common", LogStash::Timestamp.new( open_time ) )
#event.set( "@timestamp_common", open_time )
# Add 15 minutes:
curr_time = open_time + 900
i += 1
# Middle slices.
while curr_time < close_time
clone = event.clone
clone.set( "runidxseq", runidx.to_s + "_" + i.to_s )
clone.set( "@timestamp_common", LogStash::Timestamp.new( curr_time ) )
new_event_block.call( clone )
i += 1
curr_time += 900
end
# Last slice.
clone = event.clone
clone.set( "runidxseq", runidx.to_s + "_" + i.to_s )
clone.set( "@timestamp_common", LogStash::Timestamp.new( close_time ) )
#clone.set( "@timestamp_common", close_time )
new_event_block.call( clone )
'
} # /ruby
```
Further question, however: when I try to put this in it's own .rb file it won't work due to the change in environment (namely, it complains about new\_event\_block). How would it be possible to make it work as a .rb file? This is just an additional 'nice to have' as I prefer my Ruby code to be .rb files rather than jammed into the .conf but inline is fine b/c it works.
Thank you for the help!
nick
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 18, 2024, 3:59pm UTC](https://discuss.elastic.co/t/splitting-records-with-start-and-end-times-into-separate-documents-with-a-timestamp-every-x-minutes/360997/7 "2024-07-18T15:59:26Z")
> [@nickbarry](#):
>
> How would it be possible to make it work as a .rb file?
When you use a script file in a ruby filter it returns an array of events. The ruby code above lets the event object flow through and calls new\_event\_block for each clone. You would have to change
```
event.set( "@timestamp_common", LogStash::Timestamp.new( open_time ) )
```
to
```
event.set( "@timestamp_common", LogStash::Timestamp.new( open_time ) )
events = []
events << event
```
and then replace each `new_event_block.call( clone )` call with
```
events << clone
```
and finally add
```
events
```
as the last line of the code to make that the return value of the function.