Can I use Elasticsearch to save time-based events and get the intervals between them?

[gabipetrovay (Gabriel Petrovay)]

I am receiving time-based events like the following:

In March I get:

{
    "id": "e0efc334-eeb5-4740-b983-e2e8f78d8c39",
    "timestamp": "2020-03-20T05:30:00Z",
    "state": "UP",
    "service": {
        "id": "7704b2b1-02d8-4b7c-9578-667e9ceb46a2",
        "name": "service1"
    }
}

In May I get:

{
    "id": "2990fa46-77c9-4d98-a395-86561c8c07ca",
    "timestamp": "2020-05-10T04:30:00Z",
    "state": "DOWN",
    "service": {
        "id": "7704b2b1-02d8-4b7c-9578-667e9ceb46a2",
        "name": "service1"
    }
}

Can I use ES to query/aggregate the event index such that I get answers to some of the following questions:

1. What is the state of service1 now? Or at any point in time?
2. What are the intervals in which service1 was UP?
3. Can I use the date_histogram and calendar_interval aggregation that interpolates the data for the intervals missing events? (e.g. the month 2020-04 does not have any events, hence it is assumed to be the latest event received before 2020-04, which is the 2020-03-20T05:30:00Z event)

[warkolm (Mark Walkom)]

You should be able to use https://www.elastic.co/guide/en/elasticsearch/reference/7.6/transform-apis.html (and https://www.elastic.co/guide/en/elasticsearch/reference/7.6/transforms.html) for this. It's store the data into another index though.