Comparing field values with value from external file without dictionary k/v?

[hueyg]

I need to compare a field against a simple text file that contains nothing but a column of values. If the field and any value in the text file match then I need to set a field value to true, otherwise false.

It appears I can do this with the translate filter, but it appears that the file it is using for comparing requires that it is constructed in a key/value type setup.

Say I had a field called food, and in my text file are the following entries:

carrot
green beans
spinach

So if food equals "spinach" then my new field "is_vegetable" would be set to true and otherwise false.

Is that possible with translate or is my ONLY option to append a value after each entry as in:

carrot,true
green beans,true
spinach,true

I just need a simple boolean comparison and result. This is an issue because the file is going to change weekly and will have about 50,000 entries in it. Which may be another issue, but having to edit that file every time is going to be a problem.

Thanks!

[Badger]

You could do it with a ruby filter. In the init code, use 'a = File.readlines("file.txt")' to read the file into an array, then convert it to a hash using something like

@veggies = a.map {|x| [x,true]}.to_h

Then for each event test whether '@veggies.include? event.get("food")'

You could use .include? on the array, but on average that involves iterating over half the members. It is much faster on a hash.

[hueyg]

Thanks Badger,

I know next to nothing concerning Ruby and even less when it comes to calling Ruby from within a logstash filter. Understand what you are saying, so will try to fumble through it as it appears that is my only option within Logstash.

Thanks for your help as always.