Keeping a Constant variable across Log events?

Elastic Stack Logstash
1

Hello,

I want to have a constant value available for all the events getting processed by Logstash.
So for example, I want to calculate the unique session IDs based on the Epoch time of the first event. Then subsequent events will be the addition of these base epoch time and the sessions ID which are unique for a day.

[2015-Nov-18 08:49:11.968159 -0600]: <info> L3Z3-02 L3Z3-02 Starting...
[2015-Nov-18 08:50:24.431304 -0600]: <ui> L3Z3-02 0 2 Bilingual NORMAL TOUCH TIMELINE SET_STATE ECMS:atrocity/10 STATE:SINGLE_ATTRACT

Want to calculate the session ID of the UI event = base Epoch time from the INFO event + 2

Here is my code:

if [logLevel] == "info" and [status] == "Starting" {
 ruby {
     code => "require 'date'
        epochBase = DateTime.parse(event['timestamp']).to_time.to_i
        epochBase = (epochBase*10000000)/60"
     add_field => { "epochBase" => "%{epochBase}" }
  }
}

Then for other UI events I am trying to use the epochBase field like so:

ruby {
    # Recalcualting the session IDs EpochTime/60
    code => "
        epochTime = event['epochBase'] + event['sessionID'].to_i
        event['sessionIdentifier'] = epochTime.to_s"
}

But I know this will not work because the epochBase does not persist across events. Is there any way around this?

Thanks to the community here.

2

I am open to any other suggestions on the solution to this problem. Thanks

3

There is probably a more direct approach, but logically this is how I imagine it:

You want an epoch time that is unique each day and are shared among all events of that day. So to calculate your base epoch, only take the %d/%m/%Y and discard the rest.

irb(main):002:0> require 'date'
=> true
irb(main):003:0> now = DateTime.now()
=> #<DateTime: 2015-11-27T13:49:00-05:00 ((2457354j,67740s,918374000n),-18000s,2299161j)>
irb(main):005:0> today = now.strftime("%d/%m/%Y")
=> "27/11/2015"
irb(main):006:0> epoch_base = DateTime.strptime(today, "%d/%m/%Y").to_time.to_i
=> 1448582400
irb(main):007:0> puts epoch_base
1448582400
=> nil
irb(main):008:0>

4

Thanks for the reply.

Yes I can do this, but I do not want to do this for every event that comes across in the pipeline. Is there any way to do this once and make the value available for events?

5

I doubt it. Once an event has passed through LS, LS can no longer reference it (not in way I've used anyway).

6

Maybe you can use aggregate filter :
When start event occurs, you compute session I'd, and you store it in aggregate map until the end of the session in the logs.
At the session end (in the logs), you delete the map, using end_of_task=true

1 Like
7

Sure, I will give this a try. Thanks for the tip!!