Tunning Watcher

Elastic Stack Elasticsearch
1

I have created the below watcher that looks in my proxy index, it currently works however I would like to tune it.
Would like to only send events to the SIEM that exceed 20 hits for "deviceAction:TCP_AUTH_REDIRECT" from a single source IP within a specified time, lets say 10 minutes. I am new to creating watchers, any ideas on how I can make this happen or if this is even possible?

{
"trigger": {
"schedule": {
"interval": "1h"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": "xxxxxx",
"body": {
"query": {
"filtered": {
"query": {
"query_string": {
"query": "deviceAction:TCP_AUTH_REDIRECT"
}
}
}
},
"filter":{
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-1h"
}
}
}
],
"must_not": []
}
}
}
}
}
},
"condition": {
"compare": { "ctx.payload.hits.total":{
"gt": 0
}
} },
"actions": {
"cefpost": {
"webhook": {
"scheme": "http",
"host": "yyyyyy",
"port": zzzzz,
"method": "post",
"path": "/WatcherToSyslogWebservice/post",
"params": {
"watch_id": "{{ctx.watch_id}}",
"priority": "5"
},
"headers": {
"Content-Type": "application/json"
},
"body": "{{ctx.payload.hits}}"
}
}
}
}

2

Hey,

so the main question here is, how to create a query, that reflects what you need. From what I understand you need to search for all documents

  • that have a timestamp from now-10m
  • that are deviceAction:TCP_AUTH_REDIRECT

for those documents you need to aggregate on their sourceIp. And when you find aggregates, that have a count higher than 20 hits, those need to be returned. This works with the min_doc_count parameter of the terms aggregation. Also make sure you read the terms agg docs, especially the approximation part

Based on that info you can then run your watch.

Hope this helps.

--Alex

3

So trying again to get this watch to work. My ultimate goal now is to look through all source IP addresses, gather a list of all srcIPs for a period of time (let's say 15 minutes) that have more than X (maybe 150) number of "TCP_AUTH_REDIRECT" in the proxy logs "deviceAction" field.

Now, look through that list of srcIPs and if they have at least one successful GET (successful means you'll see a "TCP_NC_MISS" or "TUNNELED" in the "deviceAction" field then exclude those IPs. Those left are sources that I want to review/display.

Here is my watcher:

{
"trigger": {
"schedule": {
"interval": "15m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": "myindex",
"body": {
"size": 0,
"query": {
"filtered": {
"query": {
"query_string": {
"query": "deviceAction:TCP_AUTH_REDIRECT"
}
},
"filter":{
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-15m"
}
}
}
],
"must_not": [
{"term":{"requestUrlHost":"api.bing.com"}}
],
"filter": {
"range": {
"@timestamp": {
"from": "{{ctx.trigger.scheduled_time}}||-15m",
"to": "{{ctx.trigger.triggered_time}}"
}
}
}
}
}
}
},
"aggs": {
"aggrs": {
"terms": {
"field": "sourceAddress",
"min_doc_count": 150
}
}
}

		}
	}
		}
		},

"condition": {
"script": "ctx.payload.hits.total > 1"
},

"actions": {
"cefpost": {
"webhook": {
"scheme": "http",
"host": "xxxxx",
"port": 8080,
"method": "post",
"path": "/WatcherToSyslogWebservice/post",
"params": {
"watch_id": "{{ctx.watch_id}}",
"priority": "5"
},
"headers": {
"Content-Type": "application/json"
},
"body": ""
}
}
}
}

4

Hey,

First, can you use code blocks for better readability, see this markdown documentation for help.

And second and as already written in my last post: Is the query actually returning the data you need? If so, where is the current issue? If not, lets focus on that first.

--Alex

5

That is what I need help with, I think I need to build a multiple query but can't figure out how to do it....

Watcher should look through all my data, within a specified amount of time, say 15m, if it sees in the field deviceAction multiple TCP_AUTH_REDIRECT, lets now say 10 consecutive and at no point does it see in the deviceAction field TCP_NC_MISS or TUNNELED then I want it to be reported to me.

Currently my the query only reports sources that have more than 150 TCP_AUTH_REDIRECT in the deviceAction field within a 15 min window. Need to figure out a way to not report those that have a TCP_NC_MISS or TUNNELED within the same specified time window, they could still have 150 TCP_AUTH_REDIRECTS but won't report because TCP_NC_MISS or TUNNELED was seen.

6

Hey,

you might be better off asking this in the Elasticsearch forum, as there are much more people to help you. Keep in mind, that Elasticsearch searches on a per document base, when asking (and maybe come up with a concrete example, like some example documents). You might change your indexing strategy/document modeling strategy to support this. You might be able to use a terms aggregation to get all the deviceAction fields during that time grouped per IP to actually exclude a certain ip/set of documents that have been indexed.

--Alex