How to create ruby class variables mapped to a given ID?

Elastic Stack Logstash
1

Hi,

I'm using below Ruby filter to find the elapsed time between a start and end event for a given user. The code works fine with the events coming in sequence. But, I have event logs coming from multiple users. I have used the file_ID condition to avoid mismatch between users. But, I'm missing out on few events as the variable gets updated if multiple start events come together before the corresponding end events.

Ruby Filter

ruby {
          code => "
    		if event.get('Event') == 'Start'
    			@save_the_timestamp = event.get('@timestamp')
                @save_the_fileID = event.get('File_ID')
    		elsif event.get('Event') == 'End' && event.get('File_ID') == @save_the_fileID
    			event.set('reqtime', event.get('@timestamp') - @save_the_timestamp)
    		end
    	"
}

Is there any way I can use a persistent Ruby variable for a given user and use it.
Appreciate any help. Thanks in advance.

2

Use an aggregate filter.

3

I tried using that at the first place but I faced lot of issues as I have few other conditions as well apart from the one i mentioned above.

Isn't there a solution in Ruby to have a variable mapped to a specific ID like we have map in aggregate? If so, that would be really helpful

4

Sure, you can maintain a hash of hashes, where the outer hash is keyed by the ID, and the inner hash contains variables. That's what the map is in an aggregate filter. Just make sure you address the synchronization and threading issues that an aggregate filter addresses.

5

Hi @Badger,

Apologies for the delay, I have a quick followup question on the above solution. I tried the below ruby filter for creating a hash keyed to an File_ID field.

Ruby Filter

ruby {
		init => "@save_the_timestamp = Hash.new"
		code => "
			if event.get('Event') == 'Start'
				save_the_fileID = event.get('File_ID')
				@save_the_timestamp[save_the_fileID] = event.get('@timestamp')
			elsif event.get('Event') == 'End'
				if @save_the_timestamp[event.get('File_ID')]
                    event.set('reqtime', event.get('@timestamp') - @save_the_timestamp[save_the_fileID])
					@save_the_timestamp.delete(event.get('File_ID'))
				end
			end
		"
	}

But, I am receiving the below error

[2020-06-04T23:28:46,051][ERROR][logstash.filters.ruby ][main] Ruby exception occurred: can't convert nil into an exact number

Could you please have a look and let me know whats going wrong here?

TIA

6

Is it possible an End arrived before a Start? Have you set pipeline.workers 1? Have you disabled java_execution?

7

No, for a given File_ID, End always appears after the corresponding Start event. Also, in the End condition I am anyways checking for the existence of the hash before performing any action.

And Yes, I have done both. Disabled Java_execution & have set pipeline.workers to 1

8

You do not appear to have set save_the_fileID

9

Yes, I haven't set that. It's working fine now. Thanks a lot.
I believe this should now work with any number of hashes right?

10

I believe so.

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.