Need a Logstash filter to query a rest api/tcp on conditional statement and return JSON response

[stcdarrell (Darrell Miller)]

hi, i'd love some help, i've googled all i know to google, read the logstash documents.. and i'm still not finding what i need.

i am bringing in a ton of logs from various sources dealing with network traffic. i'd like to have a rule that based on some conditions:
-- if a field is an ipv4 ip address AND NOT an internal/private ip address send a query to an outside server to query if its a known bad/blacklisted IP or a good/whitelisted IP.

i THINK that can be done with the

if (conditions would go here) {
  filter { 
    TCP {
          
    } #end TCP
  } #end filter 
} # end if

the curl command have running against the server would be something like this:
curl -H "Authorization:22de00b5e876adwhateverwhateverwhatever" -GET "http://XXX.XXX.XXX.XXX:5000/indicators/?q=@{indicator}"

and it works and returns a JSON response with the info i need.

Am I on the right track here? i cant seem to find anyone doing anything like this.. maybe i'm not searching for the right thing.. any help or links to examples would be greatly appreciated.

[Badger]

You could use a cidr filter to add a tag if it matches private/internal networks, then test for the presence of the tag.

ETA: You don't need to roll your own with a tcp filter you should be able to use an http filter with url, query, and headers options.

[stcdarrell (Darrell Miller)]

thank you, i didnt realize there was a cidr filter, that makes sense to use that, then tag based on it.

what is the proper way to convert that curl statement i listed above into a filter {TCP {} } statement?
i cant find any good examples of that.

any help would be appreciated

[Badger]

As I said, I would suggest using an http filter rather than a tcp filter.

[stcdarrell (Darrell Miller)]

Badger thank you very much for your help.
i'm about 80% there.

I've got logstash communicating to my server.

the query i need to run is this: (using curl from linux command prompt)
curl -H "Authorization:AuthTokenGoesHere" -GET "http://192.168.4.102:5000/indicators/?q=@{indicator}"

this curl statement works.. but i'm not having very much luck translating that into the HTTP format:

filter {
        if "externalAddress" in [tags] {
                rest {
                        request => {
                                url => "http://192.168.4.102:5000/indicators/?q=%{[source][address]}"
                                method => "post"
                        } #end request
                        auth {
                                Authorization => "AuthTokenGoesHere"
                        }
                target => "cif"
                json => "true"
                } #end rest
        } # end if
} # end filter

any suggestions would really appreciated. I cant seem to find any examples of this in the discussion forums or with google

thank you
Darrell

[Badger]

I would have used the standard http plugin rather than installing the rest filter.

    http {
        url => "http://192.168.4.102:5000/indicators/"
        query =>  { "q" => "%{[source][address]}" }
        verb => POST
        headers => { "Authorization" => "AuthTokenGoesHere" }
    }