Watcher Chained Input and Painless Script

[adonahoe (Andrew)]

Hello All,

I am wondering if it is possible to query one index based on the results of another query from a different index? For instance I have added my basic watcher below with a chained input. Not entirely sure how this should work but basic I want to iterate over all the results returned in the first chained input 'BLACKLIST-IP' and query for those IP's in the REAL-DATA index. If true then return that IP into another index called RESULTS. Below is my input config thus far.
"input": {
"chain": {
"inputs": [
{
"BLACKLIST-IP": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"black-ip"
],
"types": [],
"body": {
"query": {
"match": {
"type": "ip-src"
}
}
}
}
}
}
},
{
"REAL-DATA": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"real-data"
],
"types": [],
"body": {
"query": {
"match": {
"script": {
"script": {
"lang": "painless",
"inline": "for (artifact in ctx.payload.BLACKLIST_IP.hits.hits) return artifact"(search for this artifact in the REAL-DATA index)
}
}
}
}
}
}
}
}
}
]
}
},

[Christian_Dahlqvist]

It sounds like this could be a very large chained query that it may make sense to break up somehow. Can you explain exactly what you are trying to achieve at a high level so that we can see if there are any alternate ways (apart from brute force approach) to achieve this?

[adonahoe (Andrew)]

Thank you for the response. I am basically trying to compare two indices. One contains my blacklist IPs/domains/etc. The other contains my core network information. As new blacklist IPs come in I want to check them against the last 24 hours of network data. Then every 24 hours I want to take all of the new blacklist IPs and check them against network data that is 1-6 days old to determine if I get any hits. Aka I want to do real-time checks and retroactive checks within the last week for new hits.

Then any hits I do get I want to put them into a new index called 'Hits' or something of that nature.

Thanks for the help!