I have a security product with a vendor provided ELK stack included. One of the things I'm trying to get my head around is trend dashboards over a broad period of time.
Security reports are run daily but there could be re-runs and missing runs. Each run produces a single elasticsearch record per device and there's a risk score within this data.
Initially I'm just looking to get an aggregated view of risk on a monthly basis but given the duplicate and/or missing data, a simple sum of risk across the month is going to significantly skew the visualisation making it meaningless.
The question is how to best approach this with elasticsearch/kibana. I think I need some form of aggregate query to only return the last entry in the month for each device but I can't yet get my head around how to do this in ES. My thought is with this query, handling different graphing types should be relatively straightforward.
Any pointers on how to solve this or background reading that would point me in the right direction. As a new ELK user, I'm also wondering whether there's a more ELK-centric way of thinking about the problem?