Compare Two Indices using Watcher

Elastic Stack Elasticsearch
1

Hello All,

I was wondering how I could possible compare two indices using watcher? For instance lets say I have one index with firewall data and I have another index that contains a blacklist. I was wondering how I could possibly compare all IPs in the firewall data index against all the blacklist IPs in the blacklist_IP index?

Any ideas would be great!

Thanks!

2

Hi,
sure is possible to compare two index. I do it with input chain:
https://www.elastic.co/guide/en/watcher/current/input.html#input-chain
and then u can evaluate it with painless scripts.

Cheers
pts0

1 Like
3

Thank you for the response! I did see input-chain and was wondering how hard is that to actually implement and whether or not you have a possible example you could show?

Currently I am using a python script to query between the two indices which seems to work fairly well but I would like to make a more pure solution which does not rely on a python script to do my matching for me.

Was wondering what people thought about each approach and which approach would be more scalable/easier to implement.

4

Hi,
I don't have now the example here, but if you post more detail is not a big deal to get it build.
Python script could be ok too, but is not running on elastic i imagine.
I think watcher would be easier to maintain and scale/monitor

5

Thanks for the response! Below is an example of two basic mappings for the index blacklist and the index firewall. How would I go about creating a simple watcher to compare all of the IPs in the blacklist index against the ones in the firewall index?

PUT blacklist
{
"mappings": {
"bl": {
"_all": {
"enabled": false
},
"properties": {
"B-IP": {
"type": "text"
}
}
}
}
}

PUT firewall
{
"mappings": {
"fw": {
"_all": {
"enabled": false
},
"properties": {
"SRC-IP": {
"type": "text"
}
}
}
}
}

6

Hey,

I am not sure what 'comparing' means in this context exactly. If you have some ips in your blacklist, you could either put those into the watch metadata or try to have a chained input that queries for the blacklist first and for the data second - again this highly depends on your use-case and data model, so it's hard to give a useful answer here.

--Alex

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.