Return all events between two events

[rage5474 (Raphael Geissler)]

I have a lot of log messages in one ElasticSearch index.
These log messages containing two special messages. One start message and one end message (simply having a field with the value 'START' and 'STOP').

Can I create a query in ElasticSearch that returns all logs in that index between these two messages?

Here an example that shows simplified how my data look like:

PUT log
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "mappings": {
    "_doc": {
      "properties": {
        "timestamp": {
          "type": "date",
          "format": "epoch_millis"
        },
        "msg": {
          "type": "text"
        }
      }
    }
  }
}

PUT log/_doc/1
{
    "timestamp": 1000,
    "msg" : "TO_EARLY"
}


PUT log/_doc/2
{
    "timestamp": 2000,
    "msg" : "START"
}

PUT log/_doc/3
{
    "timestamp": 3000,
    "msg" : "Hello, World!"
}

PUT log/_doc/4
{
    "timestamp": 4000,
    "msg" : "STOP"
}

PUT log/_doc/5
{
    "timestamp": 5000,
    "msg" : "TO_LATE"
}

So my question is: How does the query looks like to get the message between the START and the STOP message?

Thanks for your help.

Raphael

[dadoonet (David Pilato)]

I don't think you can do that easily. Documents are not "connected" together, meaning that they don't have any relationship.

What you should do is probably 2 queries. The first to get the min and max value (with min and max agg) for timestamp when msg is either START or STOP.
Then use those 2 values as the input of the next query: a range query from min to max.

[rage5474 (Raphael Geissler)]

Thanks for your answer.

I want to use this for a chart visualization in Kibana. Therefore I need only the values between the marker events. Is there a possibility to do this with two queries?

[dadoonet (David Pilato)]

In Kibana? Then that's a question for #kibana forum I guess.
I think it's not possible. It's always better anyway to compute everything you can at index time. Like computing a full event instead of small non related sub events...