addanuj
June 12, 2018, 8:57am
1
Hi i am trying to achieve below
for example :-
Index 2: checkpoint
using below watch condition i am only able to check only first value, how to use a loop to check all values
"condition": {
"compare": {
"ctx.payload.first.hits.hits.0._source.srcip": {
"eq": "{{ctx.payload.second.hits.hits.0._source.indicator-ip}}"
}
}
},
@richcollier Please help
addanuj
June 12, 2018, 8:59am
2
First Chain Query
{
"watch":
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logstash-checkpoint*"
],
"types": [],
"body": {
"aggs": {
"time": {
"terms": {
"field": "@timestamp",
"order": {
"_count": "desc"
}
},
"aggs": {
"srcip": {
"terms": {
"field": "srcip.keyword",
"order": {
"_count": "desc"
}
},
"aggs": {
"dstip": {
"terms": {
"field": "dstip.keyword",
"order": {
"_count": "desc"
}
}
}
}
}
}
}
},
"version": true,
"_source": {
"excludes": []
},
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
"@timestamp"
],
"query": {
"bool": {
"must": [
{
"match_all": {}
},
{
"match_all": {}
},
{
"exists": {
"field": "srcip.keyword"
}
},
{
"exists": {
"field": "dstip.keyword"
}
},
{
"range": {
"@timestamp": {
"gte": "{{ctx.trigger.scheduled_time}}||-1d",
"lte": "{{ctx.trigger.scheduled_time}}",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
}
}
}
}
}
},
addanuj
June 12, 2018, 9:02am
3
Second Chain Query
{
"second": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logstash-security-*"
],
"types": [],
"body": {
"aggs": {
"time": {
"terms": {
"field": "@timestamp",
"size": 5,
"order": {
"_count": "desc"
}
},
"aggs": {
"indicator": {
"terms": {
"field": "indicator.keyword",
"size": 5,
"order": {
"_count": "desc"
}
}
}
}
}
},
"version": true,
"_source": {
"excludes": []
},
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
"@timestamp",
"security_log.time"
],
"query": {
"bool": {
"must": [
{
"match_all": {}
},
{
"match_all": {}
},
{
"bool": {
"should": [
{
"match_phrase": {
"type.keyword": "threatintel"
}
}
],
"minimum_should_match": 1
}
},
{
"match_phrase": {
"indicator_parent": {
"query": "ip"
}
}
},
{
"exists": {
"field": "indicator"
}
},
{
"range": {
"@timestamp": {
"gte": "{{ctx.trigger.scheduled_time}}||-1d",
"lte": "{{ctx.trigger.scheduled_time}}",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
}
}
}
}
}
}
]
}
},
"condition": {
"compare": {
?????????????
},
"actions": {
"log": {
"logging": {
"level": "info",
"text": "what condition should i write"
}
}
}
}
jamesspi
June 13, 2018, 8:57am
5
Hi @addanuj ,
I don't think it's possible to loop through hits. However, I would do this differently.
I would use the ip from the 1st input, and then build the query for the second input using that variable.
Then, the compare condition would need to be based on the result of the second input. You don't even need aggregations in this case really.
So the query for your second input would be something like:
"body": {
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "type:threatintel AND indicator-ip:{{ctx.payload.first.hits.hits.0._source.srcip}}"
}
}
]
}
},
And the condition:
"condition": {
"compare": {
"ctx.payload.second.hits.total": {
"gt": 0
}
}
},
Thanks,
jamesspi
June 13, 2018, 9:03am
6
jamesspi:
And the condition:
And I should mention - you should probably be doing the comparison at the entry level (I assume you're using logstash?) using the translate filter plugin. Then just create a watch to see if there is a match. What I usually do is create a boolean field with the translate plugin, and then just watch for value=true. Much more efficient.
addanuj
June 13, 2018, 9:07am
7
Can you provide an example for this
addanuj
June 13, 2018, 9:33am
8
Hi James,#ctx .payload.first.aggregations.srcip.buckets}}{{key}} {{/ctx.payload.first.aggregations.srcip.buckets}}"
this works for me but i need to know how shall i print matching values only in action seciton
jamesspi
June 13, 2018, 9:40am
9
You won't need to compare all values, you just need to know when there is a match, correct?
Here is a link for some translate examples:
Data enrichment provides additional insights into your data, and is frequently used in Security use cases. Enrich your data with blacklists, asset inventories, and detect malware.
I've already recommended in a different thread that you should look at this post for ideas:
Ok - after a little research, this could be done in probably two ways
Method 1 - a script on the condition to see if there's an intersection of results from both queries
"condition": {
"script": "def second_results = ctx.payload.second.hits.hits.stream().map(hit->hit._source.partition_field_value).collect(Collectors.toList()); return ctx.payload.first.hits.hits.stream().map(hit -> hit._source.partition_field_value).filter(p->second_results.contains(p)).collect(Collectors.toList()).si…
Why do you keep on asking? I'm confused...
addanuj
June 13, 2018, 3:12pm
11
Thanks James, for guiding me to the right way. i will try to use logstash method also, but your answer helped me to solve some specific problems anyway,
I am really thankful to this forum and you guys, providing great support to community.
spinscale
June 22, 2018, 2:04pm
13
Using a painless script and a chained input that executes both of hte searches you mentioned in the top might be another solution.
Be aware that something like joins do not work though!
system
July 20, 2018, 2:05pm
14
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.