# Detecting variance in data

**URL:** <https://discuss.elastic.co/t/detecting-variance-in-data/79392>\
**Category:** Elasticsearch\
**Created:** [March 21, 2017, 11:01am UTC](https://discuss.elastic.co/t/detecting-variance-in-data/79392 "2017-03-21T11:01:23Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cylindric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cylindric/32/5660_2.png) [@Cylindric](https://discuss.elastic.co/u/Cylindric)\
**Post date:** [March 21, 2017, 11:01am UTC](https://discuss.elastic.co/t/detecting-variance-in-data/79392/1 "2017-03-21T11:01:23Z")

</div>

I'm not sure if this is an ES question or a Kibana question, or both, or Timelion.

We track the reputation of all our IP addresses by performing various blacklist lookups every day, and storing that data in ES.

In the most simplistic case, I just record this every day:

```
{
  "timestamp": "2017-03-20T04:23:15.0772879+00:00"
  "ip": "1.1.1.1",
  "score": 100
}

```

That means I can create pretty graphs for reputation trends per IP or range, and show all IPs that are in a danger zone. What I _cannot_ seem to do however, is detect changes. For example, some of these IPs are used by our customers, and some of those will cheefully keep tarnishing the reputation of the IPs, so it's normal to see a bunch always low. That would mean I have a big list of "danger" IPs, but many of those would be "normal".

Can I easily create a visualisation showing "all IPs that are bad now, that weren't bad yesterday"? Or over different time periods?

At the moment I am experimenting with this: when I store the new reputation I lookup the previous reputation from ES, and store that again

```
{
  "timestamp": "2017-03-21T04:23:15.0772879+00:00"
  "ip": "1.1.1.1",
  "score": 10,
  "previous_score": 100
}

```

But this bakes in the time element, so if I wanted to do "two days ago" or "a week ago" I'd have to know in advance, and lookup those additional documents on each update to update the `previous1`, `previous2` and `previous7` fields. Seems messy, is there a better way?

Ideally this would end up resulting in something I can use with Watcher to send an email out when an IP "goes bad".

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [March 21, 2017, 11:27am UTC](https://discuss.elastic.co/t/detecting-variance-in-data/79392/2 "2017-03-21T11:27:37Z")

</div>

> Can I easily create a visualisation showing "all IPs that are bad now, that weren't bad yesterday"?

Is it enough for you to record a single `lastStatusChangeDate` which is the date an IP "flips" status from good to bad (or perhaps vice-versa?). Or do you need to record the various gradations of bad?  
Querying docs where currentStatus = bad and where lastStatusChangeDate is in a time range seems easy on the querying side of things,

---

<div class="post-metadata">

**Author:** ![Cylindric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cylindric/32/5660_2.png) [@Cylindric](https://discuss.elastic.co/u/Cylindric)\
**Post date:** [March 21, 2017, 11:30am UTC](https://discuss.elastic.co/t/detecting-variance-in-data/79392/3 "2017-03-21T11:30:56Z")

</div>

That might indeed be sufficient. There are different bands of 'bad', but I could still just store a single `lastStatusChangedDate`, and I guess what the last reputation was - I'll need to know if it was heading up or down I think.

It won't scale so well once we start adding in more reputation sources, there are many to chose from, but perhaps I'll have to cross that bridge when we get to it.

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [March 21, 2017, 12:03pm UTC](https://discuss.elastic.co/t/detecting-variance-in-data/79392/4 "2017-03-21T12:03:15Z")

</div>

> [@Cylindric](#):
>
> It won't scale so well once we start adding in more reputation sources,

Large scale lookups are always expensive so some techniques that are worth considering:

1. Remove random disk seeks  
Read both your threat index(es) and your latest log data indexes sorted by IP using the `scroll` API to interleave threats and your log sightings data without the need for many individual searches.
2. Cheap/fuzzy matching  
Summarise the threat index information in a compact form optimised for local quick look-ups that may produce false positives but never false negatives - like a Bloom filter. Being primarily IP address info it may be possible to roll up threat info to whole address ranges which are marked as bad if any one IP in that range is bad. This gives a fail-fast filter that hopefully can tag 90% of your log data as good if it fails to match anything. For the 10% that get a hit on this, mark the log data as "potentially risky" and only perform additional lookups/checks on the data that gets this far. This is the same principle as a HashSet calling ".equals" on an object only after it matches the ".hashCode" rough equality check.
3. Cache threat look-up results  
Site visitors will have bursts of activity so avoid hitting Elasticsearch with each access to lookup IPs on threat indexes by caching lookup results in an LRU cache. Elasticsearch has some query caching of results but you can save a network hop from your client if you maintain your own cache of results.
4. Bulk lookup requests  
Use msearch api to batch groups of elasticsearch threat index lookups into single requests.

---

<div class="post-metadata">

**Author:** ![Cylindric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cylindric/32/5660_2.png) [@Cylindric](https://discuss.elastic.co/u/Cylindric)\
**Post date:** [March 21, 2017, 12:07pm UTC](https://discuss.elastic.co/t/detecting-variance-in-data/79392/5 "2017-03-21T12:07:16Z")

</div>

I don't think that will be a problem, this is not for inbound IP reputation checks or blocking suspicious traffic, but us keeping an eye on the reputation of _our_ IP addresses that we use for sending email etc. So the lookups are done daily, at most.

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [March 21, 2017, 12:28pm UTC](https://discuss.elastic.co/t/detecting-variance-in-data/79392/6 "2017-03-21T12:28:59Z")

</div>

Ah OK - that makes life somewhat simpler.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 18, 2017, 12:28pm UTC](https://discuss.elastic.co/t/detecting-variance-in-data/79392/7 "2017-04-18T12:28:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.