Getting last document per specific field

Elastic Stack Elasticsearch
1

Hi,

I'm collecting logs from multiple devices, and I'm trying to retrieve the last log from each device.
For instance, I'd like to have :

  • "deviceId": 4568 => {content of last log according to timestamp for this device}
  • "deviceId": 7865 => {content of last log according to timestamp for this device}

I've tried to make it work with collapse and inner hits, but it doesn't seem to return what I'm looking for.

Current attempt:

GET data-*/_search
{
  "size": 10, 
  "query": {
    "exists": {
      "field": "deviceId"
    }
  },
  "collapse": {
    "field": "deviceId.keyword",
    "inner_hits": {
      "name": "most_recent",
      "size": 1,
      "sort": [ 
        {"@timestamp": {"order": "desc", "missing" : "_last"} }
        ]
    }
  }
}
2

Can you compile a small reproducible example to try out locally with 4-5 documents? Also the Elasticsearch version you are using would be helpful.

Thanks!

3

Thanks for the quick response!
I'm using the 7.16.3 version of Elasticsearch.

Here are 5 documents:

PUT /test
{
  "mappings":{
    "properties": {
      "deviceId" : { "type" : "text" },
      "@timestamp": {"type" : "date"},
      "channel": {"type": "text"}
    }
  }
}

POST test/_doc/1
{
  "deviceId" : "456",
  "@timestamp" : "2022-03-24T11:00:00.000Z",
  "channel" : "A"
}

POST test/_doc/2
{
  "deviceId" : "456",
  "@timestamp" : "2022-03-24T12:00:00.000Z",
  "channel" : "B"
}

POST test/_doc/3
{
  "deviceId" : "123",
  "@timestamp" : "2022-03-24T11:00:00.000Z",
  "channel" : "C"
}

POST test/_doc/4
{
  "deviceId" : "123",
  "@timestamp" : "2022-03-24T12:00:00.000Z",
  "channel" : "D"
}

POST test/_doc/5
{
  "deviceId" : "123",
  "@timestamp" : "2022-03-24T13:00:00.000Z",
  "channel" : "E"
}

I want my query to return both documents below:

{
  "deviceId" : "456",
  "@timestamp" : "2022-03-24T12:00:00.000Z",
  "channel" : "B"
}
{
  "deviceId" : "123",
  "@timestamp" : "2022-03-24T13:00:00.000Z",
  "channel" : "E"
}
4

I changed your sample to not map the deviceId as sorting is not possible on text fields. Then the following looks like it worked:

GET test/_search
{
  "collapse": {
    "field": "deviceId.keyword"
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.