Compare if hosts/feeds are missing from last week

Elastic Stack Elasticsearch
1

We have 1000's of hosts (windows/linux/solaris/network devices) coming to ELK cluster. We wanted to understand if there the hosts/devices missed sending the data.

I could think of below options

  1. Get list of hosts coming consistently in past 1 week & compare against today. Trouble is, it is very intensive. Is there any metadata for just the hostnames & type of hosts rather than querying the entire indices?
  2. Is there any good Machine Learning (ML) based use-cases or watchers we can make use of to find such anomalies?
2

You could use a transform to build an entity centric index around your incoming data. In the simplest version you could have a last_seen field. That way it's easy to find hosts that disappeared with simple queries on the transform index.

This is basically a light version of your solution 1, a continuous transform only queries for the changes required. If you are only interested in the last state, you can limit the query to avoid expensive lockups on cold storage.

The other option: You use Elastic agents and use our in-build solution uptime for this.

3

thanks Hendrik.
Any example of using transform in such a manner?
Does that mean I have to put transform for every single pipeline incoming?

Surely, can this be made a more generic SIEM use-case or detection as almost all customers will face this issue?

4

The documentation should cover a lot. If you need some easy to start I recommend the webinar recording. This might not cover your use case, but gives you a general idea about transform.

Your might be interested in the painless examples, e.g. for getting the last state.

This is hard to answer, as I do not know your use cases. What do you mean with "single pipeline"? If a pipeline is a different use case than the answer is yes, if pipeline means a single host, no. Maybe you can elaborate on what you are looking for.

As said, if you look for a solution, it might be worth to use at elastic agents. We are making investments in this area, e.g. agent is using transform internally.

It seemed to me you already have data coming in, maybe you can talk a bit about what your data looks like and what you want as output.

I thought about this flow:

  • a transform that captures the last state of every host/device together with a timestamp this host has been seen the last time (this is simply a max aggregation on host)
  • a watcher that queries the transform dest index and provides alerting (if you don't need an alerting system, it might already be sufficient to create a dashboard on top of the transform dest index)

Sorry if I am answering too technical and maybe hard to consume. You might get better answers if you contact our sales and talk to a solution architect, which might have more hands down/solution based answers.

2 Likes
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.