Logstash - how to get yesterday's date

Elastic Stack Logstash
1

Hello,
I have ELK stack installed and would like to store csv data there. The only problem I am facing is that my data doesn't contain date information (just time). Example message looks something like:
aaa;bbb;ccc;23:58:5;;;ddd;;;;;;

I would like to get time field, merge it with generated date and then convert to @timestamp. The problem is that data arrives to logstash with some delay (few minutes), so it could happen that log:
aaa;bbb;ccc;23:58:5;;;ddd;;;;;;
will be processed next day. Then I need to add right date of course.

My idea is as below (written in pseudocode):
if current_hour < event_hour { // i.e. 0 < 23
event_date = yesterday_date
} else {
event_date = today_date
}

But how to get yesterday's date?

2

So before you want to make a calculation you have the @timestamp that LS received the data in the input.
You should use the event timestamp because should you ever use a persistent queue (LS PQ or Kafka) the time of filter processing maybe a long while after the event receive time.

As you noted in the pseudocode, you will never receive an event from the future :slight_smile:.
I am assuming you plan to use the ruby filter to do the date maths.
Here are some Ruby time maths for the ruby filter:

ingest_time = event.timestamp.time # get the event timestamp as a Ruby Time instance
midnight = ingest_time.to_date.to_time
# pluck the first 3 values from the Time as an array, zip in array of seconds multipliers and reduce the zip by multiplying
ingest_seconds = ingest_time.to_a[0,3].zip([1,60,3600]).reduce(0){|sum, (l,r)| sum + (l * r)}
# get the event time string, '23:57:55' use the field name that contains this value
event_time_string = event.get("[event_time]")
# split the string on ':', map the to_i method over the elements, reverse and zip reduce as above
event_seconds = event_time_string.split(':').map(&:to_i).reverse.zip([1,60,3600]).reduce(0){|sum, (l,r)| sum + (l * r)}
# set the delta from midnight
delta = event_seconds
# does the event look like its from the future
# make the delta the negative amount of seconds to add to from midnight
if event_seconds > ingest_seconds
  delta = (-24 * 3600) + event_seconds
end
event_time = midnight + delta
# use the event.timestamp API directly.
event.timestamp = LogStash::Timestamp.new(event_time)

Good Luck

3

Make sure that all events are recorded in UTC timezone and Logstash is run an ENV where TZ=UTC. The code above is not Daylight saving switch over safe.

4

Hello guyboertje

Can you help me with this. I'm using logstash 5.6

this one is working to get the current date.

      ruby {
      code => "
      event.set('dateko', Time.now.strftime('%Y-%m-%d'))
      "
      }

this one is not for yesterday's date

    ruby {
code => "
event.set('datemo', DateTime.yesterday.strftime('%Y-%m-%d'))
"
    }

with error

[2017-12-04T06:20:50,922][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method yesterday' for DateTime:Class [2017-12-04T06:20:50,926][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined methodyesterday' for DateTime:Class
[2017-12-04T06:20:50,927][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method yesterday' for DateTime:Class [2017-12-04T06:20:50,928][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined methodyesterday' for DateTime:Class
[2017-12-04T06:20:50,929][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method yesterday' for DateTime:Class [2017-12-04T06:20:50,931][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined methodyesterday' for DateTime:Class
[2017-12-04T06:20:50,934][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `yesterday' for DateTime:Class

5

Hi,
thanks for tips/suggestions. This is how I solved my issue:

    ruby {
            code => "
                    event_hour = event.get('time').split(':')[0].to_i
                    current_hour = Time.now.strftime('%H').to_i
                    if current_hour < event_hour
                            event.set('event_timestamp', (Time.now - 86400).strftime('%Y-%m-%d') + ' ' + event.get('time'))
                    else
                            event.set('event_timestamp', Time.now.strftime('%Y-%m-%d') + ' ' + event.get('time'))
                    end
            "
    }
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.