Structure for object history tracking

Elastic Stack Elasticsearch
1

hi guys,

I'm totally new working with ES and I need your help with document's structure.

We have ~100k objects (pc, laptops, servers, phones, iot etc.). Our celery workers crawl info about them from various sources (ipam, fw, dns etc) and should store this info into ES. If object is found somewhere record should be added to the elastic search with current timestamp, e.g:

POST localhost:9200/srv_monitor/_ep/ea56e270-67f4-4a5a-90ed-3f98e71056c5
{
    "name" : "srv.local.ny",
    "type" : "rhel7",
    "last_seen" : "2018.08.02T14:00:00+02:00",
    "history" : ["2018.08.02T14:00:00+02:00", "2018.08.02T13:00:00+02:00", "2018.08.01T22:00:00+02:00"]
}

Question: what is the best way of storing history (timestamps) for each object? Append them to history array or there is some more intelligent way?
Another point that we want to have history for two months only, no more. How to remove then old records from each's object history array :slight_smile: ?

All your ideas and help are welcome

regards,

2

This design is what I would call an entity-centric index (the entity being a server or other form of device).
Before we get into that have you already determined that a more traditional event-centric store is unsuitable for your queries? An event-centric store would have individual log records ("device X did action Y at timestamp Z) and you'd use queries/aggregations to summarise actions.
Some questions are easy to answer on an event-centric index and some require an entity-centric index.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.