Logstash - Filter events according to data from a REST API

[Yor_On]

Hi,

I've scoured the internet for someone with the same issue as I, but I haven't had much success in finding any precedent on getting what I need done.

I have a REST API that returns a response which looks like:

GET something/services/allowed
returns:
{
    "services" : [
    {
    "name" : "foo",
    other fields....
    },
    {
    "name" : "bar",
    other fields....
    },
    etc.
    ] 
}

And all the logs written to my Logstash look like:

{
    "service_name": "foo",
    "message" : "something",
    etc.
}

I write logs to an index based off the service_name field present in all logs. I want to make sure only services that are "authorised" to write logs get their logs into my ES cluster.

So I would like to be able to drop a log if it's service_name value does not appear as one of the names in the services list that the REST API returns

I was wondering if this is possible using the http filter or something similar?

Would greatly appreciate any help, thanks!

[Badger]

You could make the REST call once for each event, but that is going to make it expensive to process each event.

If the set of services returned by the call is constant then I would suggest making the call outside of logstash and creating a file that has a list of acceptable service names, then using a translate filter to determine whether the service_name in the current event is in that list (you would drop the event if the destination ends up set to the fallback value).