Logstash - how to get yesterday's date

Hello,
I have ELK stack installed and would like to store csv data there. The only problem I am facing is that my data doesn't contain date information (just time). Example message looks something like:
aaa;bbb;ccc;23:58:5;;;ddd;;;;;;

I would like to get time field, merge it with generated date and then convert to @timestamp. The problem is that data arrives to logstash with some delay (few minutes), so it could happen that log:
aaa;bbb;ccc;23:58:5;;;ddd;;;;;;
will be processed next day. Then I need to add right date of course.

My idea is as below (written in pseudocode):
if current_hour < event_hour { // i.e. 0 < 23
event_date = yesterday_date
} else {
event_date = today_date
}

But how to get yesterday's date?

So before you want to make a calculation you have the @timestamp that LS received the data in the input.
You should use the event timestamp because should you ever use a persistent queue (LS PQ or Kafka) the time of filter processing maybe a long while after the event receive time.

As you noted in the pseudocode, you will never receive an event from the future :slight_smile:.
I am assuming you plan to use the ruby filter to do the date maths.
Here are some Ruby time maths for the ruby filter:

ingest_time = event.timestamp.time # get the event timestamp as a Ruby Time instance
midnight = ingest_time.to_date.to_time
# pluck the first 3 values from the Time as an array, zip in array of seconds multipliers and reduce the zip by multiplying
ingest_seconds = ingest_time.to_a[0,3].zip([1,60,3600]).reduce(0){|sum, (l,r)| sum + (l * r)}
# get the event time string, '23:57:55' use the field name that contains this value
event_time_string = event.get("[event_time]")
# split the string on ':', map the to_i method over the elements, reverse and zip reduce as above
event_seconds = event_time_string.split(':').map(&:to_i).reverse.zip([1,60,3600]).reduce(0){|sum, (l,r)| sum + (l * r)}
# set the delta from midnight
delta = event_seconds
# does the event look like its from the future
# make the delta the negative amount of seconds to add to from midnight
if event_seconds > ingest_seconds
  delta = (-24 * 3600) + event_seconds
end
event_time = midnight + delta
# use the event.timestamp API directly.
event.timestamp = LogStash::Timestamp.new(event_time)

Good Luck

Make sure that all events are recorded in UTC timezone and Logstash is run an ENV where TZ=UTC. The code above is not Daylight saving switch over safe.

Hello guyboertje

Can you help me with this. I'm using logstash 5.6

this one is working to get the current date.

      ruby {
      code => "
      event.set('dateko', Time.now.strftime('%Y-%m-%d'))
      "
      } 

this one is not for yesterday's date

    ruby {
code => "
event.set('datemo', DateTime.yesterday.strftime('%Y-%m-%d'))
"
    }

with error

[2017-12-04T06:20:50,922][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method yesterday' for DateTime:Class [2017-12-04T06:20:50,926][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined methodyesterday' for DateTime:Class
[2017-12-04T06:20:50,927][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method yesterday' for DateTime:Class [2017-12-04T06:20:50,928][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined methodyesterday' for DateTime:Class
[2017-12-04T06:20:50,929][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method yesterday' for DateTime:Class [2017-12-04T06:20:50,931][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined methodyesterday' for DateTime:Class
[2017-12-04T06:20:50,934][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `yesterday' for DateTime:Class

Hi,
thanks for tips/suggestions. This is how I solved my issue:

    ruby {
            code => "
                    event_hour = event.get('time').split(':')[0].to_i
                    current_hour = Time.now.strftime('%H').to_i
                    if current_hour < event_hour
                            event.set('event_timestamp', (Time.now - 86400).strftime('%Y-%m-%d') + ' ' + event.get('time'))
                    else
                            event.set('event_timestamp', Time.now.strftime('%Y-%m-%d') + ' ' + event.get('time'))
                    end
            "
    }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.