Detecting variance in data

Elastic Stack Elasticsearch
1

I'm not sure if this is an ES question or a Kibana question, or both, or Timelion.

We track the reputation of all our IP addresses by performing various blacklist lookups every day, and storing that data in ES.

In the most simplistic case, I just record this every day:

{
  "timestamp": "2017-03-20T04:23:15.0772879+00:00"
  "ip": "1.1.1.1",
  "score": 100
}

That means I can create pretty graphs for reputation trends per IP or range, and show all IPs that are in a danger zone. What I cannot seem to do however, is detect changes. For example, some of these IPs are used by our customers, and some of those will cheefully keep tarnishing the reputation of the IPs, so it's normal to see a bunch always low. That would mean I have a big list of "danger" IPs, but many of those would be "normal".

Can I easily create a visualisation showing "all IPs that are bad now, that weren't bad yesterday"? Or over different time periods?

At the moment I am experimenting with this: when I store the new reputation I lookup the previous reputation from ES, and store that again

{
  "timestamp": "2017-03-21T04:23:15.0772879+00:00"
  "ip": "1.1.1.1",
  "score": 10,
  "previous_score": 100
}

But this bakes in the time element, so if I wanted to do "two days ago" or "a week ago" I'd have to know in advance, and lookup those additional documents on each update to update the previous1, previous2 and previous7 fields. Seems messy, is there a better way?

Ideally this would end up resulting in something I can use with Watcher to send an email out when an IP "goes bad".

2

Can I easily create a visualisation showing "all IPs that are bad now, that weren't bad yesterday"?

Is it enough for you to record a single lastStatusChangeDate which is the date an IP "flips" status from good to bad (or perhaps vice-versa?). Or do you need to record the various gradations of bad?
Querying docs where currentStatus = bad and where lastStatusChangeDate is in a time range seems easy on the querying side of things,

3

That might indeed be sufficient. There are different bands of 'bad', but I could still just store a single lastStatusChangedDate, and I guess what the last reputation was - I'll need to know if it was heading up or down I think.

It won't scale so well once we start adding in more reputation sources, there are many to chose from, but perhaps I'll have to cross that bridge when we get to it.

4

Large scale lookups are always expensive so some techniques that are worth considering:

  1. Remove random disk seeks
    Read both your threat index(es) and your latest log data indexes sorted by IP using the scroll API to interleave threats and your log sightings data without the need for many individual searches.
  2. Cheap/fuzzy matching
    Summarise the threat index information in a compact form optimised for local quick look-ups that may produce false positives but never false negatives - like a Bloom filter. Being primarily IP address info it may be possible to roll up threat info to whole address ranges which are marked as bad if any one IP in that range is bad. This gives a fail-fast filter that hopefully can tag 90% of your log data as good if it fails to match anything. For the 10% that get a hit on this, mark the log data as "potentially risky" and only perform additional lookups/checks on the data that gets this far. This is the same principle as a HashSet calling ".equals" on an object only after it matches the ".hashCode" rough equality check.
  3. Cache threat look-up results
    Site visitors will have bursts of activity so avoid hitting Elasticsearch with each access to lookup IPs on threat indexes by caching lookup results in an LRU cache. Elasticsearch has some query caching of results but you can save a network hop from your client if you maintain your own cache of results.
  4. Bulk lookup requests
    Use msearch api to batch groups of elasticsearch threat index lookups into single requests.
5

I don't think that will be a problem, this is not for inbound IP reputation checks or blocking suspicious traffic, but us keeping an eye on the reputation of our IP addresses that we use for sending email etc. So the lookups are done daily, at most.

6

Ah OK - that makes life somewhat simpler.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.